-- Logs begin at Thu 2022-10-06 03:52:30 UTC, end at Thu 2022-10-06 04:58:58 UTC. -- Oct 06 04:17:42.991487 dv-cinder-860227-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 06 04:17:43.000127 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Oct 6 04:17:42 2022] *** Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: nodename: dv-cinder-860227-2-iscsi Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: machine: x86_64 Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: clock source: unix Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: pcre jit disabled Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: detected number of CPU cores: 4 Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: current working directory: / Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: detected binary path: /usr/bin/uwsgi-core Oct 06 04:17:43.002527 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: your processes number limit is 47680 Oct 06 04:17:43.003130 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: your memory page size is 4096 bytes Oct 06 04:17:43.003130 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: detected max file descriptor number: 2048 Oct 06 04:17:43.003130 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: lock engine: pthread robust mutexes Oct 06 04:17:43.003130 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: thunder lock: enabled Oct 06 04:17:43.003130 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 06 04:17:43.003130 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 06 04:17:43.042515 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: Python main interpreter initialized at 0x556bdf1c69c0 Oct 06 04:17:43.042515 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: python threads support enabled Oct 06 04:17:43.042515 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: your server socket listen backlog is limited to 100 connections Oct 06 04:17:43.042515 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: your mercy for graceful operations on workers is 90 seconds Oct 06 04:17:43.042845 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: mapped 403077 bytes (393 KB) for 2 cores Oct 06 04:17:43.042936 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: *** Operational MODE: preforking *** Oct 06 04:17:43.043205 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: *** uWSGI is running in multiple interpreter mode *** Oct 06 04:17:43.043205 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: spawned uWSGI master process (pid: 81389) Oct 06 04:17:43.043395 dv-cinder-860227-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 06 04:17:43.044212 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: spawned uWSGI worker 1 (pid: 81399, cores: 1) Oct 06 04:17:43.044312 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: spawned uWSGI worker 2 (pid: 81400, cores: 1) Oct 06 04:17:43.046118 dv-cinder-860227-2-iscsi devstack@keystone.service[81389]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 06 04:17:47.353234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 06 04:17:47.353234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: from cryptography.utils import int_from_bytes Oct 06 04:17:47.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 06 04:17:47.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: from cryptography.utils import int_from_bytes Oct 06 04:17:47.357929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 06 04:17:47.357929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: from cryptography.utils import int_from_bytes Oct 06 04:17:47.358397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 06 04:17:47.358397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: from cryptography.utils import int_from_bytes Oct 06 04:17:47.404651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 06 04:17:47.404835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 06 04:17:47.405013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] command line args: [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 06 04:17:47.405279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 06 04:17:47.405456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ================================================================================ {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 06 04:17:47.405706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] admin_token = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.405947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] config_dir = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.406133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.406312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] config_source = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.406431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 06 04:17:47.406507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] debug = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.406637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 06 04:17:47.406726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.406952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] command line args: [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 06 04:17:47.407110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.407195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 06 04:17:47.407257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ================================================================================ {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 06 04:17:47.407326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.407496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] admin_token = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.407605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.407735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.407846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] config_dir = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.407942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] list_limit = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] config_source = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] log_config_append = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] debug = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] log_dir = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] log_file = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.408975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] list_limit = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] log_config_append = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.409963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] log_dir = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] log_file = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410393 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.410995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.411984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.33/identity {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] publish_errors = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.412993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] standard_threads = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.413929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.33/identity {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] publish_errors = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] use_journal = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.414892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] use_json = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] use_stderr = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] use_syslog = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] standard_threads = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.415920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.416040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.416162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.416392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.416518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.416642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.416642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] use_journal = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.416858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.416971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] use_json = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.417091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.417207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] use_stderr = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.417329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.417445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] use_syslog = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.417564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.417685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 06 04:17:47.417800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.417914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.418874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.419946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.420893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.421791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.422513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.422643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.422767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.422883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.423853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.424882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.425946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.426868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.427999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.428901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429813 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.429911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.430864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.431989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.432936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.433971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.434905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435393 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.435974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.436928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.437048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.437163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.437278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.437391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.437506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.437631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.438332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.438431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.438553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.438668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.438782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.438905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.439957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.440913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.441994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442346 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] auth.external = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.442948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] auth.password = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] auth.token = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.443941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] auth.external = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.444887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] credential.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] auth.password = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445709 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] auth.token = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.445950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.446887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] credential.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.447988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.448934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.449993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.450925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.451897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.452971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.453977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.454100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.454217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.454360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.454507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.454609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.454726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.454841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.455986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.456922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.457980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.458915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.459879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.460939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.461995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.462119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.462270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.462386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.462502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.462618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.462732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.462856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.463940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.464987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.465903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466365 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.466971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467813 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.467938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.468991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.469905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.470994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.471945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.472889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.473954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474184 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.474894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] role.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] role.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.475968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.476967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.477910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] role.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] role.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.478977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.479915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.480965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.481892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.482942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.483935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.484058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.484175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.484291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.484406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.485982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.486918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.487051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.487167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.487282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.487409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.487529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.487900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.connection = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.488976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.489895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.490988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.connection = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.491931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492393 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.492989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81400) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 06 04:17:47.493702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81399) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 06 04:17:47.519466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.519695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.519920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.520607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.520842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.521078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.528203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.529379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.556511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.556735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.557464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.557695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.609195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.610457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.658199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.658428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.660011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.660249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.814741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.814908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.815807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.838602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.838779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.838870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839365 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.839998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.840125 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.851523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.851939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.864348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.864579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.864741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.864841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.864983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81400) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.865820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.866031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81399) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 06 04:17:47.882696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.882905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 06 04:17:47.883130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.883271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.883357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 06 04:17:47.883470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 06 04:17:47.883674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.883756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.883875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 06 04:17:47.883988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.884147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.884238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.884410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.884493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.884668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.884927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.894066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.894341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.894565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.894650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.894847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.894931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.895147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.895429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.901840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.902375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.904744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.905028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.905307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.905388 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.905586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.905669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.905868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.905951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.906126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.906248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.907832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.908003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.908088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.908289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.908556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.930720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.931367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.932220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.932501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.932841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.933131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.937164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.937446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.938238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.938375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.938751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.939056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.944593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.944876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.945409 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.945699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.950556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.950853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.951164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.951448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.951530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.951720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.951816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.951994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.952077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.952263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.952346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.952530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.952615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.952803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.953073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.953408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.968208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.968486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.969344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.969678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.972827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.973635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.976017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.976312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.976864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.977162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.980477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.981327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.983560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.984420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:47.987073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.987372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.987906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.988150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.988234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.988452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.988537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.988727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.988810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.989012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.989293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:47.989572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.005068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.005353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.005616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.005792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.005875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.006082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.006352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.006619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.012699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.013486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.014242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.015058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.015869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.016139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.016646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.016925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.018907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.019217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.019499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.019581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.019781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.019863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.020131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.020214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.020420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.020817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.031820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.032109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.032386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.032664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.032745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.033013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.033103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.033297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.033390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.033581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.033665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.033961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.034249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.034529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.049576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.050598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.052496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.053550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.055404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.056453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.056844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.057907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.058285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.058575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.058999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.059395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.059680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.060094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.065494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.066587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.066892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.068100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.069807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.070196 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.070480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.070755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.071014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.071414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.071702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.071978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.078657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.078987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.079259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.079946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.080247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.080530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.086113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81400) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.086397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.086686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.086982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.087262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.087443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81399) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 06 04:17:48.087524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.087733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.087826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.088919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.089111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.089278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.089397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.089677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.089756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.089948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.090052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81400) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.090355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.090686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.090967 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.091249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81399) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 06 04:17:48.125546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x556bdf1c69c0 pid: 81400 (default app) Oct 06 04:17:48.126582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x556bdf1c69c0 pid: 81399 (default app) Oct 06 04:17:48.128865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38af2d-ac82-4737-a82e-1a649ce88d9d None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:48.129148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38af2d-ac82-4737-a82e-1a649ce88d9d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:48.129423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca38af2d-ac82-4737-a82e-1a649ce88d9d None None] PATH_INFO: `/v3` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:48.133550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1/1] 10.222.0.33 () {50 vars in 791 bytes} [Thu Oct 6 04:17:48 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:17:54.942537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5058d4e-5bae-4107-900e-9570ef137e6a None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:54.943085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5058d4e-5bae-4107-900e-9570ef137e6a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:54.943085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5058d4e-5bae-4107-900e-9570ef137e6a None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:54.947278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1/2] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:17:54 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:17:54.952748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:54.953335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:54.953433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:55.025293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: INFO dbcounter [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] Registered counter for database keystone Oct 06 04:17:55.044758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG oslo_db.sqlalchemy.engines [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81400) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 06 04:17:55.051733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writer thread running {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 06 04:17:55.235968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG passlib.handlers.bcrypt [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81400) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 06 04:17:55.236358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG passlib.handlers.bcrypt [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81400) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 06 04:17:55.295459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:17:55.297115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:17:55.424142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 2/3] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:17:54 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 472 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 06 04:17:55.870245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5058d4e-5bae-4107-900e-9570ef137e6a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:17:55.939479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: INFO dbcounter [None req-b5058d4e-5bae-4107-900e-9570ef137e6a None None] Registered counter for database keystone Oct 06 04:17:55.960855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG oslo_db.sqlalchemy.engines [None req-b5058d4e-5bae-4107-900e-9570ef137e6a None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81399) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 06 04:17:55.966928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writer thread running {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 06 04:17:56.087834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5058d4e-5bae-4107-900e-9570ef137e6a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:17:56.092094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cb0ccf-c51d-4687-a49f-94b18728761b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:17:56.092726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb0ccf-c51d-4687-a49f-94b18728761b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:56.093007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb0ccf-c51d-4687-a49f-94b18728761b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:56.093279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cb0ccf-c51d-4687-a49f-94b18728761b admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:56.101918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cb0ccf-c51d-4687-a49f-94b18728761b admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:17:56.154189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f6cb0ccf-c51d-4687-a49f-94b18728761b admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 06 04:17:56.154470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cb0ccf-c51d-4687-a49f-94b18728761b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:17:56.164466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 2/4] 10.222.0.33 () {58 vars in 1324 bytes} [Thu Oct 6 04:17:55 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:17:56.171229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:17:56.188906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff17dc3-2c84-4d66-8e7f-cb621bc15874 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:17:56.193549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abe3ca6-e434-4144-802e-721a264b9583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:17:56.194627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abe3ca6-e434-4144-802e-721a264b9583 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:56.194905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abe3ca6-e434-4144-802e-721a264b9583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:56.195203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abe3ca6-e434-4144-802e-721a264b9583 admin admin] PATH_INFO: `/v3/projects/7760d317949542fd9d86e529eca476cc` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:56.196742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abe3ca6-e434-4144-802e-721a264b9583 admin admin] RBAC: Authorizing `identity:get_project(project_id=7760d317949542fd9d86e529eca476cc)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:17:56.251133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING keystone.common.rbac_enforcer.enforcer [None req-3abe3ca6-e434-4144-802e-721a264b9583 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 06 04:17:56.251473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abe3ca6-e434-4144-802e-721a264b9583 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:17:56.256113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 3/5] 10.222.0.33 () {56 vars in 1207 bytes} [Thu Oct 6 04:17:56 2022] GET /identity/v3/projects/7760d317949542fd9d86e529eca476cc => generated 344 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:17:58.360608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33415172-88ca-492a-9b1a-a2506b9e0200 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:58.361155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33415172-88ca-492a-9b1a-a2506b9e0200 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:58.361155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33415172-88ca-492a-9b1a-a2506b9e0200 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:58.365621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 3/6] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:17:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:17:58.371092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd790f4-6130-469a-9b61-635f8cd3a0ae None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:58.371509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd790f4-6130-469a-9b61-635f8cd3a0ae None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:58.371638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd790f4-6130-469a-9b61-635f8cd3a0ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:58.392669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9fd790f4-6130-469a-9b61-635f8cd3a0ae None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:17:58.394198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9fd790f4-6130-469a-9b61-635f8cd3a0ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:17:58.401879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 4/7] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:17:58 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 06 04:17:58.673041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33415172-88ca-492a-9b1a-a2506b9e0200 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:17:58.684935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33415172-88ca-492a-9b1a-a2506b9e0200 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:17:58.689833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80879c72-fff9-4f3a-b400-15adbf792c55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:17:58.690429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80879c72-fff9-4f3a-b400-15adbf792c55 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:58.690708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80879c72-fff9-4f3a-b400-15adbf792c55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:58.691012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80879c72-fff9-4f3a-b400-15adbf792c55 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:58.698687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80879c72-fff9-4f3a-b400-15adbf792c55 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:17:58.700468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80879c72-fff9-4f3a-b400-15adbf792c55 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:17:58.709116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 4/8] 10.222.0.33 () {58 vars in 1324 bytes} [Thu Oct 6 04:17:58 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:17:58.714341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd790f4-6130-469a-9b61-635f8cd3a0ae None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:17:58.724126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd790f4-6130-469a-9b61-635f8cd3a0ae None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:17:58.728463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d5406c-d424-44cf-a547-6cdbf1fd1c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:17:58.729086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d5406c-d424-44cf-a547-6cdbf1fd1c4f admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:17:58.729368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d5406c-d424-44cf-a547-6cdbf1fd1c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:17:58.729647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d5406c-d424-44cf-a547-6cdbf1fd1c4f admin admin] PATH_INFO: `/v3/users/f6cec6cd063d4a0684ad9f283f12b406` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:17:58.730952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d5406c-d424-44cf-a547-6cdbf1fd1c4f admin admin] RBAC: Authorizing `identity:get_user(user_id=f6cec6cd063d4a0684ad9f283f12b406)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:17:58.732814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d5406c-d424-44cf-a547-6cdbf1fd1c4f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:17:58.735338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 5/9] 10.222.0.33 () {56 vars in 1198 bytes} [Thu Oct 6 04:17:58 2022] GET /identity/v3/users/f6cec6cd063d4a0684ad9f283f12b406 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:00.941094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0686c0-a289-443f-a11d-309f234c0297 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:00.941660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0686c0-a289-443f-a11d-309f234c0297 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:00.941660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0686c0-a289-443f-a11d-309f234c0297 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:00.942953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 5/10] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:00.948367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef583c2f-b95f-4aca-868e-352841b9eb80 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:00.948800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef583c2f-b95f-4aca-868e-352841b9eb80 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:00.949051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef583c2f-b95f-4aca-868e-352841b9eb80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:00.951129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55eaa697-2642-47bd-a33f-5b2338f69041 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:00.951416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55eaa697-2642-47bd-a33f-5b2338f69041 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:00.951689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55eaa697-2642-47bd-a33f-5b2338f69041 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:00.953561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 6/11] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:00.958851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:00.959164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:00.959438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:00.967856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ef583c2f-b95f-4aca-868e-352841b9eb80 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:00.969704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ef583c2f-b95f-4aca-868e-352841b9eb80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:01.003070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG passlib.handlers.bcrypt [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81399) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 06 04:18:01.003450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG passlib.handlers.bcrypt [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81399) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 06 04:18:01.049900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 6/12] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:01.052685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:01.054260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:01.154529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 7/13] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:01.317241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef583c2f-b95f-4aca-868e-352841b9eb80 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.341273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:01.420900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.428627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef583c2f-b95f-4aca-868e-352841b9eb80 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:01.462266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:01.462886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:01.463243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:01.463465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:01.475526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:01.484797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:01.485908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:01.487227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 7/14] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:01.491941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.521817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24e219c-b6e7-480b-972f-fe04375860bb None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:01.524070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:01.559849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:01.560702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:01.560982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:01.561258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:01.572592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:01.590669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476870fb-e5db-4aec-80a9-3080e6ece735 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:01.624975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:01.625586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:01.625862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:01.626130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:01.626786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:01.627118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:01.628509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:01.630842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:01.632372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:01.639906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 8/15] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:01.646033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.730333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 8/16] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:01.736124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.739205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:01.743530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed45999-7862-4e85-92c1-d6ba6e3a614d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:01.776900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:01.777530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:01.777812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:01.778083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] PATH_INFO: `/v3/users/f6cec6cd063d4a0684ad9f283f12b406` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:01.779241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] RBAC: Authorizing `identity:get_user(user_id=f6cec6cd063d4a0684ad9f283f12b406)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:01.780785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:01.782387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 9/17] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/users/f6cec6cd063d4a0684ad9f283f12b406 => generated 247 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:01.787695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.801908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:01.836530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce423b48-301f-4897-af05-211bfa33600b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:01.869971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:01.870812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:01.871122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:01.871392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:01.885251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39cc891-5195-427d-b813-66e021ae0dd9 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:01.889302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:01.891632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:18:01.910001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:01.910935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:18:01.911801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:18:01.911801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:18:01.911801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:01.911801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 9/18] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:18:01.917025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.920040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:01.920666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:01.920938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:01.921204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:01.922659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:01.924157 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:01.925701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 10/19] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:01.931168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:01.977156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:02.010208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:02.023274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba4b7b3-aa7f-42d9-bf73-5fa7d43e9a33 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:02.029266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df719378-f678-4869-8921-cb45c55931f5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:02.056882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b9e19a-6874-4216-a22b-9c9a05bc25df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:02.057477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b9e19a-6874-4216-a22b-9c9a05bc25df None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:02.057740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b9e19a-6874-4216-a22b-9c9a05bc25df None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:02.058016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b9e19a-6874-4216-a22b-9c9a05bc25df None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:02.058645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b9e19a-6874-4216-a22b-9c9a05bc25df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:02.058943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b9e19a-6874-4216-a22b-9c9a05bc25df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:02.060347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b9e19a-6874-4216-a22b-9c9a05bc25df None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:02.063499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a1be62-5faa-4d0b-b698-347a42ffafc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:02.064098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a1be62-5faa-4d0b-b698-347a42ffafc0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:02.064375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a1be62-5faa-4d0b-b698-347a42ffafc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:02.064645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a1be62-5faa-4d0b-b698-347a42ffafc0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:02.065359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a1be62-5faa-4d0b-b698-347a42ffafc0 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.domain.id=default, user.id=f6cec6cd063d4a0684ad9f283f12b406) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:02.065661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a1be62-5faa-4d0b-b698-347a42ffafc0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:02.067114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a1be62-5faa-4d0b-b698-347a42ffafc0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:02.073704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 11/20] 10.222.0.33 () {56 vars in 1322 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/role_assignments?user.id=f6cec6cd063d4a0684ad9f283f12b406&scope.domain.id=default&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 233 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:02.078752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 10/21] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:18:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:04.578156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b3487-1a43-4434-8862-ce5e9e8be42d None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:04.578718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b3487-1a43-4434-8862-ce5e9e8be42d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:04.578718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b3487-1a43-4434-8862-ce5e9e8be42d None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:04.580546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 12/22] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:04.585590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2738a2a-e0ad-45de-8b8f-b53d92340f59 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:04.586492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2738a2a-e0ad-45de-8b8f-b53d92340f59 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:04.586983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2738a2a-e0ad-45de-8b8f-b53d92340f59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:04.615112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d2738a2a-e0ad-45de-8b8f-b53d92340f59 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:04.616810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d2738a2a-e0ad-45de-8b8f-b53d92340f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:04.645302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d25fd6-bd9a-4ad6-ad15-d1ffda75c42f None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:04.645929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d25fd6-bd9a-4ad6-ad15-d1ffda75c42f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:04.646406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d25fd6-bd9a-4ad6-ad15-d1ffda75c42f None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:04.648237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 13/23] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:04.653291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e6a90-f869-4079-9b29-da7a22b331db None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:04.653811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e6a90-f869-4079-9b29-da7a22b331db None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:04.654296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e6a90-f869-4079-9b29-da7a22b331db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:04.688814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-400e6a90-f869-4079-9b29-da7a22b331db None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:04.693046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-400e6a90-f869-4079-9b29-da7a22b331db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:04.774256 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 11/24] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:04.782767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907ecd80-1625-40fd-909c-1d1491d0b09c None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:04.783083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907ecd80-1625-40fd-909c-1d1491d0b09c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:04.783363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907ecd80-1625-40fd-909c-1d1491d0b09c None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:04.784817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 12/25] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:04 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:04.793035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37830627-e2bd-45bb-aa1f-e9c4dd406427 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:04.793284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37830627-e2bd-45bb-aa1f-e9c4dd406427 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:04.793559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37830627-e2bd-45bb-aa1f-e9c4dd406427 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:04.836884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-37830627-e2bd-45bb-aa1f-e9c4dd406427 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:04.837982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-37830627-e2bd-45bb-aa1f-e9c4dd406427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:04.841430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 14/26] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:04.913646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:04.945417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 13/27] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:05.108598 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e6a90-f869-4079-9b29-da7a22b331db None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:05.203011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37830627-e2bd-45bb-aa1f-e9c4dd406427 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:05.203870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:05.255807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e6a90-f869-4079-9b29-da7a22b331db None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:05.335669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:05.336314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:05.336594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:05.336876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:05.375862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:05.380747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:05.381612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:05.382381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 15/28] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:05.409970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37830627-e2bd-45bb-aa1f-e9c4dd406427 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:05.411992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:05.468831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:05.469477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:05.469841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:05.470155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:05.486339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:05.488129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:18:05.514581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:05.517293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:18:05.519343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:18:05.519343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:18:05.519343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:05.519343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 14/29] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:05.522811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:05.559526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:05.662459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4a164-f0fd-44f4-98c2-52abac037c7a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:05.677386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9466e0d-c9fd-4249-a031-f22ecb0484a7 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:05.701962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:05.703121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:05.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:05.703853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:05.704751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:05.705051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:05.707176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:05.716959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f762e3-127c-4ff3-8cad-9c7ff4b09c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:05.717630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f762e3-127c-4ff3-8cad-9c7ff4b09c38 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:05.717955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f762e3-127c-4ff3-8cad-9c7ff4b09c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:05.718794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f762e3-127c-4ff3-8cad-9c7ff4b09c38 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:05.719678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f762e3-127c-4ff3-8cad-9c7ff4b09c38 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:05.719987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f762e3-127c-4ff3-8cad-9c7ff4b09c38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:05.720543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 15/30] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:18:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:05.721353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:05.721466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f762e3-127c-4ff3-8cad-9c7ff4b09c38 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:05.723093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:05.731778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 16/31] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:05.733383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b20bd-68b3-4299-8b52-57619e0927db None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:05.733769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b20bd-68b3-4299-8b52-57619e0927db None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:05.734059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b20bd-68b3-4299-8b52-57619e0927db None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:05.736179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 17/32] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:05.737023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72b20bd-68b3-4299-8b52-57619e0927db None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:05.840490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:05.855060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2c8fba-db23-43dd-b36f-30f3e68ac24a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:05.882892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72b20bd-68b3-4299-8b52-57619e0927db None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:05.892848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:05.893605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:05.894005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:05.894388 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:05.895205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:05.896782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:05.935343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1dbd8d-98dd-41e7-80af-15d225496be4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:05.936175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1dbd8d-98dd-41e7-80af-15d225496be4 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:05.936618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1dbd8d-98dd-41e7-80af-15d225496be4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:05.937574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1dbd8d-98dd-41e7-80af-15d225496be4 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:05.938456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1dbd8d-98dd-41e7-80af-15d225496be4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:05.940241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1dbd8d-98dd-41e7-80af-15d225496be4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:06.030683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 16/33] 10.222.0.33 () {60 vars in 1131 bytes} [Thu Oct 6 04:18:05 2022] POST /identity/v3/roles => generated 219 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:06.032042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:06.105523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 18/34] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:18:05 2022] POST /identity/v3/projects => generated 301 bytes in 369 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:06.107470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511bd52-a7ce-43aa-b4a5-bf036f697e9b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.107470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511bd52-a7ce-43aa-b4a5-bf036f697e9b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.107897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511bd52-a7ce-43aa-b4a5-bf036f697e9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.139561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-2511bd52-a7ce-43aa-b4a5-bf036f697e9b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:06.141850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2511bd52-a7ce-43aa-b4a5-bf036f697e9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:06.169283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfaedd6-5a3d-486c-a040-ce44a7cf30c8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:06.216053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:06.217177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.218261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.218830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] PATH_INFO: `/v3/users/f6cec6cd063d4a0684ad9f283f12b406` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.220885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] RBAC: Authorizing `identity:get_user(user_id=f6cec6cd063d4a0684ad9f283f12b406)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:06.222925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:06.225145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 17/35] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:06 2022] GET /identity/v3/users/f6cec6cd063d4a0684ad9f283f12b406 => generated 247 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:06.232161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:06.261219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:06.277550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:06.332251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 19/36] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:06.337637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e001d2a-85f5-474d-8f45-a9e117edd0a7 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.338125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e001d2a-85f5-474d-8f45-a9e117edd0a7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.338454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e001d2a-85f5-474d-8f45-a9e117edd0a7 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.342884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 20/37] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:06 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:06.349864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d08d8-c05d-431c-b19b-ec366c2caad7 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.350252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d08d8-c05d-431c-b19b-ec366c2caad7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.356166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d08d8-c05d-431c-b19b-ec366c2caad7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.398280 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-4a3d08d8-c05d-431c-b19b-ec366c2caad7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:06.399586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4a3d08d8-c05d-431c-b19b-ec366c2caad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:06.432889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328fe70d-ba93-4da6-a3a4-5f6abc7baf5d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:06.479812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e115ae9-c256-4390-8ead-fd3887fecb8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:06.480774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e115ae9-c256-4390-8ead-fd3887fecb8d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.481245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e115ae9-c256-4390-8ead-fd3887fecb8d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.482359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e115ae9-c256-4390-8ead-fd3887fecb8d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.485796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e115ae9-c256-4390-8ead-fd3887fecb8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:06.487778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e115ae9-c256-4390-8ead-fd3887fecb8d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:06.489606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 18/38] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:06.491126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc64b9e8-f09e-4f8b-90c3-be574077f2e1 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.491757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc64b9e8-f09e-4f8b-90c3-be574077f2e1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.492241 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc64b9e8-f09e-4f8b-90c3-be574077f2e1 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.494047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 19/39] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:06.500379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc64b9e8-f09e-4f8b-90c3-be574077f2e1 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:06.506039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 21/40] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:06.507345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd19ea21-aac8-4e5d-a23c-e578df8c76a6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.507637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd19ea21-aac8-4e5d-a23c-e578df8c76a6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.507918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd19ea21-aac8-4e5d-a23c-e578df8c76a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.534074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dd19ea21-aac8-4e5d-a23c-e578df8c76a6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:06.535312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dd19ea21-aac8-4e5d-a23c-e578df8c76a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:06.612787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 22/41] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:06.625829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:06.643105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc64b9e8-f09e-4f8b-90c3-be574077f2e1 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:06.646126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd19ea21-aac8-4e5d-a23c-e578df8c76a6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:06.687316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:06.689014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.689895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:06.690605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.691065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] PATH_INFO: `/v3/domains/default/users/f6cec6cd063d4a0684ad9f283f12b406/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.695128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=f6cec6cd063d4a0684ad9f283f12b406, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:06.696974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:06.724737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 20/42] 10.222.0.33 () {58 vars in 1361 bytes} [Thu Oct 6 04:18:06 2022] PUT /identity/v3/domains/default/users/f6cec6cd063d4a0684ad9f283f12b406/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:18:06.787736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd19ea21-aac8-4e5d-a23c-e578df8c76a6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:06.806760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:06.830044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:06.830994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.831495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.831938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.833377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:06.835636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:06.867625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 23/43] 10.222.0.33 () {60 vars in 1131 bytes} [Thu Oct 6 04:18:06 2022] POST /identity/v3/roles => generated 225 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:18:06.887970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:06.951244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbf4a21-4c3d-41b3-bf17-cefb7546ed7b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:06.994532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:06.995188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:06.995467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:06.995738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:06.996433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:06.997853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:07.020348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 21/44] 10.222.0.33 () {60 vars in 1131 bytes} [Thu Oct 6 04:18:06 2022] POST /identity/v3/roles => generated 223 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:18:07.133279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:07.137664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0207a173-634b-4ce2-a74a-622beaad4701 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:07.182813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8812f4ac-81d3-4e30-a8dc-ca27e8862047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:07.183512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812f4ac-81d3-4e30-a8dc-ca27e8862047 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:07.183792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812f4ac-81d3-4e30-a8dc-ca27e8862047 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:07.184068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8812f4ac-81d3-4e30-a8dc-ca27e8862047 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:07.185253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8812f4ac-81d3-4e30-a8dc-ca27e8862047 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:07.186806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8812f4ac-81d3-4e30-a8dc-ca27e8862047 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:07.188936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 24/45] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:07.194824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:07.199151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:07.310595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1904f0aa-cffa-4e50-9a4c-0a250c9eed26 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:07.344249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae0d9d5-e6e8-459e-9abf-e59ffa1ab2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:07.344857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae0d9d5-e6e8-459e-9abf-e59ffa1ab2f6 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:07.345137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae0d9d5-e6e8-459e-9abf-e59ffa1ab2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:07.345414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae0d9d5-e6e8-459e-9abf-e59ffa1ab2f6 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:07.346059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae0d9d5-e6e8-459e-9abf-e59ffa1ab2f6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:07.347563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae0d9d5-e6e8-459e-9abf-e59ffa1ab2f6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:07.378703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 22/46] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:18:07 2022] POST /identity/v3/projects => generated 312 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:08.930319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b56c4b-eea8-4121-bc5d-73ce2d1c06de None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:08.930873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b56c4b-eea8-4121-bc5d-73ce2d1c06de None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:08.930873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b56c4b-eea8-4121-bc5d-73ce2d1c06de None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:08.932229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 25/47] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:08.937952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55545852-335a-4f75-b225-2b4f5a9de17d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:08.938335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55545852-335a-4f75-b225-2b4f5a9de17d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:08.938504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55545852-335a-4f75-b225-2b4f5a9de17d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:08.956922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-55545852-335a-4f75-b225-2b4f5a9de17d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:08.958410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-55545852-335a-4f75-b225-2b4f5a9de17d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:09.006595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:09.023529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 23/48] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:09.292601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b56c4b-eea8-4121-bc5d-73ce2d1c06de None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:09.393048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b56c4b-eea8-4121-bc5d-73ce2d1c06de None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:09.430072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9936c8cc-8c22-4431-9c73-396956bd79c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:09.430710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9936c8cc-8c22-4431-9c73-396956bd79c0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:09.431011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9936c8cc-8c22-4431-9c73-396956bd79c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:09.431288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9936c8cc-8c22-4431-9c73-396956bd79c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:09.442112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9936c8cc-8c22-4431-9c73-396956bd79c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-9936c8cc-8c22-4431-9c73-396956bd79c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:09.446883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:09.447750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:09.448512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 26/49] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:09.454289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55545852-335a-4f75-b225-2b4f5a9de17d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:09.522987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b9ec0c-a640-4b3d-8303-7ffe267fcd5d None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:09.523243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b9ec0c-a640-4b3d-8303-7ffe267fcd5d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:09.523526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b9ec0c-a640-4b3d-8303-7ffe267fcd5d None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:09.524874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 27/50] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:09.529917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f4c26-cc05-471b-b6f3-a4211d169c35 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:09.530199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f4c26-cc05-471b-b6f3-a4211d169c35 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:09.531112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f4c26-cc05-471b-b6f3-a4211d169c35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:09.549088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ad9f4c26-cc05-471b-b6f3-a4211d169c35 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:09.550273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ad9f4c26-cc05-471b-b6f3-a4211d169c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:09.552041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55545852-335a-4f75-b225-2b4f5a9de17d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:09.572075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:09.586280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:09.586886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:09.587196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:09.587474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:09.588138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:09.588441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:09.589819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:09.602851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 24/51] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:09.608545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:09.619116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 28/52] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:09.759646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:09.809531 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193f57a7-b9f2-4eb3-b2ca-dc9a88dddaf0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:09.841989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:09.842601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:09.842942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:09.843230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] PATH_INFO: `/v3/users/f6cec6cd063d4a0684ad9f283f12b406` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:09.844335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] RBAC: Authorizing `identity:get_user(user_id=f6cec6cd063d4a0684ad9f283f12b406)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:09.845746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:09.847344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 25/53] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:09 2022] GET /identity/v3/users/f6cec6cd063d4a0684ad9f283f12b406 => generated 247 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:09.852751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f4c26-cc05-471b-b6f3-a4211d169c35 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:09.893996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:09.954502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f4c26-cc05-471b-b6f3-a4211d169c35 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:09.992526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:09.993166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:09.995756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:09.995756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:09.995756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:09.996920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:09.998654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 29/54] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:10.001562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431d970-cde6-49f0-90b8-ed3407c72e0a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:10.004840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:10.017880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:10.029960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:10.036437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:10.037063 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:10.037343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:10.037618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:10.038725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:10.040182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:10.041747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 26/55] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:10.047900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:10.105257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30e5029-e879-4a4f-874d-630faf140928 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:10.140509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e8f0f2-11d8-4f43-94f4-7ae938e4e2d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:10.141109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e8f0f2-11d8-4f43-94f4-7ae938e4e2d6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:10.141390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e8f0f2-11d8-4f43-94f4-7ae938e4e2d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:10.141667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e8f0f2-11d8-4f43-94f4-7ae938e4e2d6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:10.142398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e8f0f2-11d8-4f43-94f4-7ae938e4e2d6 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.domain.id=default, user.id=f6cec6cd063d4a0684ad9f283f12b406) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:10.142706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e8f0f2-11d8-4f43-94f4-7ae938e4e2d6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:10.144205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e8f0f2-11d8-4f43-94f4-7ae938e4e2d6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:10.147876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a4bc1a-4eb7-4b44-92f6-7265aee6995f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:10.149814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 30/56] 10.222.0.33 () {56 vars in 1322 bytes} [Thu Oct 6 04:18:10 2022] GET /identity/v3/role_assignments?user.id=f6cec6cd063d4a0684ad9f283f12b406&scope.domain.id=default&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 530 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:10.184015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9feaa338-e422-4780-a1cf-cab7a76c0ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:10.184633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feaa338-e422-4780-a1cf-cab7a76c0ce5 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:10.184930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feaa338-e422-4780-a1cf-cab7a76c0ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:10.185187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feaa338-e422-4780-a1cf-cab7a76c0ce5 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:10.185852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9feaa338-e422-4780-a1cf-cab7a76c0ce5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:10.188050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9feaa338-e422-4780-a1cf-cab7a76c0ce5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:10.226989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 27/57] 10.222.0.33 () {60 vars in 1140 bytes} [Thu Oct 6 04:18:10 2022] POST /identity/v3/projects => generated 298 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:18:12.398737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd784c48-d921-431a-850f-c6ea2bd53611 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:12.399305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd784c48-d921-431a-850f-c6ea2bd53611 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:12.399305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd784c48-d921-431a-850f-c6ea2bd53611 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:12.400663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 31/58] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:12.406143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6890f-1585-44c9-88f4-65db9b2bef40 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:12.406576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6890f-1585-44c9-88f4-65db9b2bef40 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:12.406701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6890f-1585-44c9-88f4-65db9b2bef40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:12.424973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-97d6890f-1585-44c9-88f4-65db9b2bef40 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:12.426167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-97d6890f-1585-44c9-88f4-65db9b2bef40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:12.454402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:12.493377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 28/59] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:12.767258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd784c48-d921-431a-850f-c6ea2bd53611 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:12.855423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:12.870107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd784c48-d921-431a-850f-c6ea2bd53611 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:12.902968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cac9ce-95de-49a4-8613-018608859b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:12.903614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cac9ce-95de-49a4-8613-018608859b3a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:12.903898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cac9ce-95de-49a4-8613-018608859b3a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:12.904177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cac9ce-95de-49a4-8613-018608859b3a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:12.906027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cac9ce-95de-49a4-8613-018608859b3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:12.907634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cac9ce-95de-49a4-8613-018608859b3a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:12.909198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 32/60] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:12.914678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6890f-1585-44c9-88f4-65db9b2bef40 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:13.008398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6890f-1585-44c9-88f4-65db9b2bef40 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:13.040805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5203fc37-e237-428b-876c-48fa7616c6e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:13.041327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203fc37-e237-428b-876c-48fa7616c6e0 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:13.041602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203fc37-e237-428b-876c-48fa7616c6e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:13.041871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5203fc37-e237-428b-876c-48fa7616c6e0 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:13.042512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5203fc37-e237-428b-876c-48fa7616c6e0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:13.043976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5203fc37-e237-428b-876c-48fa7616c6e0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:13.068498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 29/61] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:18:12 2022] POST /identity/v3/users => generated 275 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:16.156273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf1a15-c4d4-4a26-9100-75a9261acc43 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.156788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf1a15-c4d4-4a26-9100-75a9261acc43 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.156788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf1a15-c4d4-4a26-9100-75a9261acc43 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.158171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 33/62] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:16.176611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf6324a-da39-4977-a4e0-239e1a214b99 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.177081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf6324a-da39-4977-a4e0-239e1a214b99 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf6324a-da39-4977-a4e0-239e1a214b99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.200445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-dcf6324a-da39-4977-a4e0-239e1a214b99 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:16.201668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dcf6324a-da39-4977-a4e0-239e1a214b99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:16.219898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b65c43a-c2b1-4a2a-a5dd-340eb191931f None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.220184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b65c43a-c2b1-4a2a-a5dd-340eb191931f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.220467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b65c43a-c2b1-4a2a-a5dd-340eb191931f None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.221945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 34/63] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:16.230974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d983753-d271-4937-b6d6-818700658661 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.231284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d983753-d271-4937-b6d6-818700658661 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.231565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d983753-d271-4937-b6d6-818700658661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.235383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:16.268007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3d983753-d271-4937-b6d6-818700658661 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:16.269562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3d983753-d271-4937-b6d6-818700658661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:16.297840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 30/64] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:16.392375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 35/65] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:16.595433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf6324a-da39-4977-a4e0-239e1a214b99 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:16.699821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d983753-d271-4937-b6d6-818700658661 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:16.734720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf6324a-da39-4977-a4e0-239e1a214b99 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:16.779719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c4c9b1-6b69-4f79-9786-1fd9c2605440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:16.780841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4c9b1-6b69-4f79-9786-1fd9c2605440 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.781112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4c9b1-6b69-4f79-9786-1fd9c2605440 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.781384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4c9b1-6b69-4f79-9786-1fd9c2605440 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.794599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c4c9b1-6b69-4f79-9786-1fd9c2605440 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-81c4c9b1-6b69-4f79-9786-1fd9c2605440 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:16.802332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:16.803263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:16.804023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 31/66] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:16.805827 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16095ff-0e8e-4eb7-a213-cda6454d6015 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.806115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16095ff-0e8e-4eb7-a213-cda6454d6015 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.806394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16095ff-0e8e-4eb7-a213-cda6454d6015 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.807672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:16.808587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 32/67] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:16.814426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16095ff-0e8e-4eb7-a213-cda6454d6015 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:16.826441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d983753-d271-4937-b6d6-818700658661 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:16.847436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:16.865024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d8e58e-fc78-4545-925e-e21a33f1edb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:16.865712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d8e58e-fc78-4545-925e-e21a33f1edb4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.865999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d8e58e-fc78-4545-925e-e21a33f1edb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.866291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d8e58e-fc78-4545-925e-e21a33f1edb4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.878678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d8e58e-fc78-4545-925e-e21a33f1edb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-e6d8e58e-fc78-4545-925e-e21a33f1edb4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:16.883587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:16.884427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:16.885184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 36/68] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:16.890813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e91042-5267-4750-863f-05c377fc87c2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.890813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e91042-5267-4750-863f-05c377fc87c2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.890813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e91042-5267-4750-863f-05c377fc87c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.907685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-17e91042-5267-4750-863f-05c377fc87c2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:16.908900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-17e91042-5267-4750-863f-05c377fc87c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:16.939933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16095ff-0e8e-4eb7-a213-cda6454d6015 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:16.981021 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 37/69] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:16.981986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e91042-5267-4750-863f-05c377fc87c2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:16.993749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:16.994356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:16.994632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:16.994904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:16.995598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:16.995897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:16.997296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:17.016813 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 33/70] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:17.022395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:17.096820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.136477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e91042-5267-4750-863f-05c377fc87c2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:17.184419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.211484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:17.211484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.211484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.211484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.212257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:17.212328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:17.213678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:17.227489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 38/71] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:17.236178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:17.239448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777e6be0-4341-4cee-b87d-3bb5d240b90c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:17.300535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e398f3-f635-4b14-83b9-3acc35879be4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:17.301205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e398f3-f635-4b14-83b9-3acc35879be4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.301436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e398f3-f635-4b14-83b9-3acc35879be4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.301713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e398f3-f635-4b14-83b9-3acc35879be4 None admin] PATH_INFO: `/v3/users/f6cec6cd063d4a0684ad9f283f12b406` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.302843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e398f3-f635-4b14-83b9-3acc35879be4 None admin] RBAC: Authorizing `identity:get_user(user_id=f6cec6cd063d4a0684ad9f283f12b406)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:17.304323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e398f3-f635-4b14-83b9-3acc35879be4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:17.310420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 34/72] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/users/f6cec6cd063d4a0684ad9f283f12b406 => generated 247 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:17.312172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dffa21-3b66-4314-a78b-ca107319a14a None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.312489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dffa21-3b66-4314-a78b-ca107319a14a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.312816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dffa21-3b66-4314-a78b-ca107319a14a None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.314056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 35/73] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:17.315073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e993922f-2a6c-4ba4-a775-386238e54112 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.315348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e993922f-2a6c-4ba4-a775-386238e54112 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.315616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e993922f-2a6c-4ba4-a775-386238e54112 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.316794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 36/74] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:17.317476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e993922f-2a6c-4ba4-a775-386238e54112 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:17.346558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef737155-1825-40f5-ac01-af2779addb0f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:17.382212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:17.382843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.383148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.383432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.391712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.398179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:17.399680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:17.401319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 39/75] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:17.406897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:17.423497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e993922f-2a6c-4ba4-a775-386238e54112 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:17.456971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7e56ca-bb68-4b9d-9e6d-0b5bad9b8cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:17.457572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e56ca-bb68-4b9d-9e6d-0b5bad9b8cee None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.457852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e56ca-bb68-4b9d-9e6d-0b5bad9b8cee None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.458124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7e56ca-bb68-4b9d-9e6d-0b5bad9b8cee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.470978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.506237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ff0c3b-558c-480b-826c-072777335f43 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:17.528174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7e56ca-bb68-4b9d-9e6d-0b5bad9b8cee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:17.529646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7e56ca-bb68-4b9d-9e6d-0b5bad9b8cee None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:17.540069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5460bd68-c852-480d-b0df-86f46bb427d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:17.540693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460bd68-c852-480d-b0df-86f46bb427d5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.540973 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460bd68-c852-480d-b0df-86f46bb427d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.541240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460bd68-c852-480d-b0df-86f46bb427d5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.551053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5460bd68-c852-480d-b0df-86f46bb427d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-5460bd68-c852-480d-b0df-86f46bb427d5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:17.555734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:17.556717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:17.557473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 40/76] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:17.558127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b70dc6-daa1-4ac1-a813-5c4f03df3fd6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.558191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b70dc6-daa1-4ac1-a813-5c4f03df3fd6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.558466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b70dc6-daa1-4ac1-a813-5c4f03df3fd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.575972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e1b70dc6-daa1-4ac1-a813-5c4f03df3fd6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:17.577730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e1b70dc6-daa1-4ac1-a813-5c4f03df3fd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:17.627442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.632297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 37/77] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:17.633671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748e020-202e-44e4-a2a8-237e9f611617 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.633959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748e020-202e-44e4-a2a8-237e9f611617 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.634234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748e020-202e-44e4-a2a8-237e9f611617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.644813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 41/78] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:17.645578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b70dc6-daa1-4ac1-a813-5c4f03df3fd6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:17.665099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e748e020-202e-44e4-a2a8-237e9f611617 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:17.666284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e748e020-202e-44e4-a2a8-237e9f611617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:17.705646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.740764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 38/79] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:17.749146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e748e020-202e-44e4-a2a8-237e9f611617 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:17.754869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b70dc6-daa1-4ac1-a813-5c4f03df3fd6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:17.792810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:17.793738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.794246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.794776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.854919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.870794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:17.872546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:17.879561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e748e020-202e-44e4-a2a8-237e9f611617 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:17.922714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:17.923284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:17.923567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:17.923842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:17.924503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:17.924858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:17.926231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:17.934651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 39/80] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:17.935390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:17.977397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:17.984773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 42/81] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:17.990537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.048964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4135c8a1-2e20-4cc4-bb4f-e8473e020837 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.073873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:18.083397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.084033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.084314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.084595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.085737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.087232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.088877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 40/82] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:18.089147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a52f60-25e6-4b67-b0f6-62fa74577aff None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.089639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.124336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.124954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.125230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.125500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.126601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.128080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.129625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 43/83] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:18.130364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.191991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8290a606-8f34-410e-b81f-cbf3d714e653 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.193495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:18.227105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.227818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.228286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.228568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.229714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.231198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.231911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7269f784-97c0-49d9-a29f-67fda7521183 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.232801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 41/84] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:18.234403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.266410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.267059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.267338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.267606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.268708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.270158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.271803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 44/85] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:18.272558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.296917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:18.339874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c8baf-428c-48c4-8dd3-3112762ec92a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.373823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.374446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.374730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.375120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.375209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a507aa5-775c-480d-8373-f648b689aa16 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.385297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:18.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:18.392114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:18.392893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 42/86] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:18.393535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.409605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.410212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.410487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.410757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.411515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=f6cec6cd063d4a0684ad9f283f12b406) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:18.411814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.413249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.418491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 45/87] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/role_assignments?user.id=f6cec6cd063d4a0684ad9f283f12b406&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:18.419282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.437601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:18.509831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd27c66-e4e1-4439-89f9-4c9b41e3114a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.524495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:18.540633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f302f70f-1500-4603-bb9c-7d34966f6118 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.549933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.550578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.550855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.551165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.551857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.553319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.579214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.579833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.579952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 43/88] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:18:18 2022] POST /identity/v3/users => generated 289 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:18.580408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.580739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.580852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.689721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.692091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.695392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:18.722028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded18c7d-6c01-45ca-be90-ca6618a85591 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.773771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.774579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.774919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.775665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.776421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=38e7ae862c514700ac27c1f36f7caf25) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:18.776999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.778991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.790176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 44/89] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/role_assignments?user.id=38e7ae862c514700ac27c1f36f7caf25&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 259 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:18.791187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.816479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:18.833794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 46/90] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:18.839711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.918034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b45583e-823a-43a7-8707-3e243289ab0e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:18.961695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:18.962321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:18.962605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:18.962883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:18.963603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] RBAC: Adding query filter params (name=member) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:18.963911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:18.965321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:18.977930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 45/91] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:18.984538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:18.985681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2b9eb-717b-4f0e-b5b9-5025509aac64 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:19.005522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:19.038395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:19.038992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:19.039291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:19.039572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:19.041197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:19.042686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:19.044796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 47/92] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:19.051688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:19.125143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a7e7cc-8f9f-429d-8585-47ec7b4e6d10 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:19.129387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:19.166326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:19.167079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:19.167367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:19.167652 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:19.168913 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:19.170361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:19.172661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 46/93] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:18 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:19.178363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:19.185088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b08c0a3-7fd1-4925-a197-d34ec4223054 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:19.238179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:19.239140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:19.239428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:19.239704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:19.241189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] RBAC: Adding query filter params (role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=38e7ae862c514700ac27c1f36f7caf25) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:19.241561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:19.243619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:19.252174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 48/94] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:19 2022] GET /identity/v3/role_assignments?user.id=38e7ae862c514700ac27c1f36f7caf25&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 259 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:19.271873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:19.297311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4d7a90-9f54-4690-95a4-db1397572a73 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:19.336335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:19.336960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:19.337301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:19.337627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:19.430909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:19.432550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:19.546518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:19.574780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 47/95] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:19.580764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:19.689236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252c2e4-6109-41f8-9cca-0b14670c6aa0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:19.707536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:19.725094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b36b137-df97-4616-9154-f17a5d2854b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:19.725735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b36b137-df97-4616-9154-f17a5d2854b4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:19.726019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b36b137-df97-4616-9154-f17a5d2854b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:19.726294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b36b137-df97-4616-9154-f17a5d2854b4 None admin] PATH_INFO: `/v3/projects/90b8410d9d4946d5ac6094702e13a192` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:19.727469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b36b137-df97-4616-9154-f17a5d2854b4 None admin] RBAC: Authorizing `identity:get_project(project_id=90b8410d9d4946d5ac6094702e13a192)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:19.728938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b36b137-df97-4616-9154-f17a5d2854b4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:19.730932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 49/96] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:19 2022] GET /identity/v3/projects/90b8410d9d4946d5ac6094702e13a192 => generated 312 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:19.737601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:19.843581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04a5288-6a66-42e9-8064-dafb162a4d0d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:19.880815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465fc3e9-2757-4453-a638-0ffc232cd5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:19.881430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465fc3e9-2757-4453-a638-0ffc232cd5b0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:19.881711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465fc3e9-2757-4453-a638-0ffc232cd5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:19.882042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465fc3e9-2757-4453-a638-0ffc232cd5b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:19.882780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465fc3e9-2757-4453-a638-0ffc232cd5b0 None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=90b8410d9d4946d5ac6094702e13a192, user.id=38e7ae862c514700ac27c1f36f7caf25) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:19.883123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465fc3e9-2757-4453-a638-0ffc232cd5b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:19.884648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465fc3e9-2757-4453-a638-0ffc232cd5b0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:19.891280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 48/97] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:19 2022] GET /identity/v3/role_assignments?user.id=38e7ae862c514700ac27c1f36f7caf25&scope.project.id=90b8410d9d4946d5ac6094702e13a192&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:22.047133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78461105-eb9e-4e37-bb09-a625f8e8bf06 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.047732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78461105-eb9e-4e37-bb09-a625f8e8bf06 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.047811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78461105-eb9e-4e37-bb09-a625f8e8bf06 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.049400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 50/98] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:22.051108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fda3d2-db2c-435d-9df3-892a97551291 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.051489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fda3d2-db2c-435d-9df3-892a97551291 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.051664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fda3d2-db2c-435d-9df3-892a97551291 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.056544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 49/99] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:22.060200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a0e8-78ec-4e58-a2a9-85df303e392c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.060561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a0e8-78ec-4e58-a2a9-85df303e392c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.060848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c5a0e8-78ec-4e58-a2a9-85df303e392c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.065751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049f1f1-1157-46f5-80e9-61bc86784d07 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.066037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049f1f1-1157-46f5-80e9-61bc86784d07 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.066317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049f1f1-1157-46f5-80e9-61bc86784d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.087222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b6c5a0e8-78ec-4e58-a2a9-85df303e392c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:22.088489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b6c5a0e8-78ec-4e58-a2a9-85df303e392c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:22.113382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c049f1f1-1157-46f5-80e9-61bc86784d07 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:22.114615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c049f1f1-1157-46f5-80e9-61bc86784d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:22.210200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:22.248189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 50/100] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:22.251027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc3febe-1b82-4554-8153-06e9544d85ca None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.251323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc3febe-1b82-4554-8153-06e9544d85ca None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.251600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc3febe-1b82-4554-8153-06e9544d85ca None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.253005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 51/101] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:22.258593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356c005e-9e8f-4fb2-95ac-c9775d7c82e9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.258870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356c005e-9e8f-4fb2-95ac-c9775d7c82e9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.263683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356c005e-9e8f-4fb2-95ac-c9775d7c82e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.292367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 51/102] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:22.304461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-356c005e-9e8f-4fb2-95ac-c9775d7c82e9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:22.305605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-356c005e-9e8f-4fb2-95ac-c9775d7c82e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:22.342816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855462df-f060-43c4-9751-8bcf1583a108 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.343192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855462df-f060-43c4-9751-8bcf1583a108 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.343399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855462df-f060-43c4-9751-8bcf1583a108 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.345034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 52/103] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:22.350758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841f279c-225e-4282-bdb9-304637fd641d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.351075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841f279c-225e-4282-bdb9-304637fd641d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.351358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841f279c-225e-4282-bdb9-304637fd641d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.379511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-841f279c-225e-4282-bdb9-304637fd641d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:22.380731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-841f279c-225e-4282-bdb9-304637fd641d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:22.463297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 52/104] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:22.509327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 53/105] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:22.675015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356c005e-9e8f-4fb2-95ac-c9775d7c82e9 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:22.826594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356c005e-9e8f-4fb2-95ac-c9775d7c82e9 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:22.829036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:22.862163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:22.863079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:22.863359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:22.863626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:22.875945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:22.880554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841f279c-225e-4282-bdb9-304637fd641d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:22.883584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:22.884455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:22.885226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:22.885811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 53/106] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:22.896434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:22.979673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.009251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d816d7a-db32-44f8-9c16-11663c567438 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.059881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.061167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841f279c-225e-4282-bdb9-304637fd641d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.062083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.062581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.063111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.064057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:23.064569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.066259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.077534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 54/107] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:23.079161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.105943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.106576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.106854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.107155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.115497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.123148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:23.129142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:23.130099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:23.130909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:23.132380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 54/108] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:23.132612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.202165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9147b7cb-9d16-4ed3-bf7f-23d8608f5f4c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.247684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.248605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.249102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.249642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.249985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee13c7d8-08a7-4bda-a031-cfc1ab45e792 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.258248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.260662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:23.265923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:23.266819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:23.267624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 55/109] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:23.268272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.290455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746efe47-fd83-465a-861e-dc53ef1de241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.291204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746efe47-fd83-465a-861e-dc53ef1de241 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.291395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746efe47-fd83-465a-861e-dc53ef1de241 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.291670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746efe47-fd83-465a-861e-dc53ef1de241 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.302737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746efe47-fd83-465a-861e-dc53ef1de241 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-746efe47-fd83-465a-861e-dc53ef1de241 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:23.308537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:23.309385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:23.310148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 55/110] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:23.311710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b453cb6-6592-43dc-b5a5-2bc62191c778 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.311826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b453cb6-6592-43dc-b5a5-2bc62191c778 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.312055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b453cb6-6592-43dc-b5a5-2bc62191c778 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.313310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 56/111] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:23.314007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b453cb6-6592-43dc-b5a5-2bc62191c778 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.358856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.374575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad65e7d-533c-455d-bffb-333dbd842dfc None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.408370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5c73b0-7b15-45e8-b5da-02426efa215c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.408988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5c73b0-7b15-45e8-b5da-02426efa215c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.409266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5c73b0-7b15-45e8-b5da-02426efa215c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.409541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5c73b0-7b15-45e8-b5da-02426efa215c None admin] PATH_INFO: `/v3/users/f6cec6cd063d4a0684ad9f283f12b406` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.410656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5c73b0-7b15-45e8-b5da-02426efa215c None admin] RBAC: Authorizing `identity:get_user(user_id=f6cec6cd063d4a0684ad9f283f12b406)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.412136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5c73b0-7b15-45e8-b5da-02426efa215c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.413717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 56/112] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/users/f6cec6cd063d4a0684ad9f283f12b406 => generated 247 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:23.414798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40542ea6-0710-415e-8582-e6dc2c88e82c None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.415129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40542ea6-0710-415e-8582-e6dc2c88e82c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.415476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40542ea6-0710-415e-8582-e6dc2c88e82c None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.416856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 57/113] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:23 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:23.416956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b453cb6-6592-43dc-b5a5-2bc62191c778 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.420567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40542ea6-0710-415e-8582-e6dc2c88e82c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.451023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.452192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.452629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.452905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.453564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:23.453869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.455298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.463313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 57/114] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:23.467875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.496743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.523566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40542ea6-0710-415e-8582-e6dc2c88e82c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.555965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e0da76-a15d-4148-91bf-67d093ce634c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.556565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0da76-a15d-4148-91bf-67d093ce634c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.556843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0da76-a15d-4148-91bf-67d093ce634c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.557113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0da76-a15d-4148-91bf-67d093ce634c None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.557762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e0da76-a15d-4148-91bf-67d093ce634c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:23.558062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e0da76-a15d-4148-91bf-67d093ce634c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.559530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e0da76-a15d-4148-91bf-67d093ce634c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.567372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 58/115] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:23.568136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce5876c-3732-45ad-8553-d26937cb96b0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.571454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f61bcdf-b9c6-4615-af16-aa8544f6a9a9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.571742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f61bcdf-b9c6-4615-af16-aa8544f6a9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.572023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f61bcdf-b9c6-4615-af16-aa8544f6a9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.590423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7f61bcdf-b9c6-4615-af16-aa8544f6a9a9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:23.591637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7f61bcdf-b9c6-4615-af16-aa8544f6a9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:23.601285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.601883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.602160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.602432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.602555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.603111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:23.603413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.604864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.613281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 58/116] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:23.614061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.663046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 59/117] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:23.663046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550feba-aa6a-4147-825d-23faa5dcfd39 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.663046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550feba-aa6a-4147-825d-23faa5dcfd39 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.663046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550feba-aa6a-4147-825d-23faa5dcfd39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.693952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0550feba-aa6a-4147-825d-23faa5dcfd39 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:23.695308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0550feba-aa6a-4147-825d-23faa5dcfd39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:23.708447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.719519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940570c6-88fa-4ad8-8433-9f785250650c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.755338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.755971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.756251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.756531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.757648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.759150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.760797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 59/118] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:23.761554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.772515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 60/119] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:23.773563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550feba-aa6a-4147-825d-23faa5dcfd39 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.870149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:23.890004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bf280a-b11c-4864-ab1b-5cffa274fc71 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.898073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550feba-aa6a-4147-825d-23faa5dcfd39 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:23.934332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.935130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.935475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.936343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.937585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.937697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:23.938359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:23.938705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:23.939072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:23.939398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.940761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:23.941325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 60/120] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:23.941936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.942252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:23.944209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 61/121] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:23.949143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:23.976381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.053823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5b788b-b487-4940-bed5-868606e5bfa4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.056908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a52153-0519-48e9-8567-d486a5dda63a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.087373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.088574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.089010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.089285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] PATH_INFO: `/v3/users/70e4d1e5029249588fd3f1a27b399a8d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.089715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.090369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.090649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.090931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/f6cec6cd063d4a0684ad9f283f12b406/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.093663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] RBAC: Authorizing `identity:create_grant(project_id=55dd5dfdb0894fdfad4138219ed4e9e8, user_id=f6cec6cd063d4a0684ad9f283f12b406, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.095123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.104757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] RBAC: Authorizing `identity:get_user(user_id=70e4d1e5029249588fd3f1a27b399a8d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.106236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.107112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 62/122] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:23 2022] PUT /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/f6cec6cd063d4a0684ad9f283f12b406/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:24.108096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 61/123] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:23 2022] GET /identity/v3/users/70e4d1e5029249588fd3f1a27b399a8d => generated 289 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:24.108197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.111902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.116681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.218402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.238900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293c8dd2-c605-4e86-9b31-fb3812ffbd98 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.239300 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72a472-7d5b-42c1-a722-92dce30ffda5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.281488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.281687 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.282183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.282464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.282741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.283209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.283671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.283972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.284299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.285472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.287305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 62/124] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:24 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:24.288856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.300885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:24.306116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:24.307016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:24.307810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 63/125] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:24.308446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.374845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.401195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ab28c6-3c10-4937-b26b-07d78f684bb1 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.419532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d047bebc-2e1c-48cc-b06f-3e4f20a7e42a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.437657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.438322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.438603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.438875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.440027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.441477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.443138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 63/126] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:24 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:24.443904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.460224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.460699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.460976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.461245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.462347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.463969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.465410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 64/127] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:24.468118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.468242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.555120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca86a69-fd8c-4874-b1fe-f2c958333c89 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.578233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd5835-41eb-4b44-a7f4-5016afd3b8bd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.590302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.590911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.591232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.591502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.610304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.616405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.617062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.617339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.617613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/38e7ae862c514700ac27c1f36f7caf25/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.619554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] RBAC: Authorizing `identity:create_grant(project_id=55dd5dfdb0894fdfad4138219ed4e9e8, user_id=38e7ae862c514700ac27c1f36f7caf25, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.620999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.636970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 65/128] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:24 2022] PUT /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/38e7ae862c514700ac27c1f36f7caf25/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:24.638570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.676816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.678493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.703906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.795554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f7d5d-f38d-4f63-bc42-e6ef8da008bb None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.803708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 64/129] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:24.804759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.847131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.847977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.848406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.848676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.849351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] RBAC: Adding query filter params (name=member) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:24.849651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.852655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.863098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 66/130] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:24.864243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.921727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742ac0b-73bf-4901-afae-5cc9e6b2ebfa None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:24.943857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.945950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:24.960068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:24.960740 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:24.961028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:24.961360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/38e7ae862c514700ac27c1f36f7caf25/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:24.963609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] RBAC: Authorizing `identity:create_grant(project_id=55dd5dfdb0894fdfad4138219ed4e9e8, user_id=38e7ae862c514700ac27c1f36f7caf25, role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:24.965081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:24.984738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 65/131] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:24 2022] PUT /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/38e7ae862c514700ac27c1f36f7caf25/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:24.985599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:24.997587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d02de8-3316-46e6-bd55-e50d3580ac22 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:25.039691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:25.040356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:25.040636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:25.040911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:25.041906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:25.044247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:25.080973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 67/132] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:18:24 2022] POST /identity/v3/projects => generated 302 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:25.082154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:25.155455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ab4333-6f6a-4454-a5ca-18b3338ed50f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:25.227097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:25.227771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:25.228246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:25.228533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:25.230733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:25.232427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:25.234377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 66/133] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:24 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:25.240900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:25.285484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd6ad86-7c97-48ba-888b-01f7f89f4604 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:25.293922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:25.324760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:25.364694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:25.365271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:25.365556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:25.365884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:25.370524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:25.370524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:25.370524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 68/134] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:25 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:25.376854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:25.416576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fbd253-7bec-41d1-8e20-aef1cab5fd65 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:25.469587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:25.470204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:25.470481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:25.470803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:25.471570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] RBAC: Adding query filter params (role.id=5cef7e0c81c54474b23931f34295ad6e, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=70e4d1e5029249588fd3f1a27b399a8d) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:25.471924 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:25.473405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:25.484191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 67/135] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:25 2022] GET /identity/v3/role_assignments?user.id=70e4d1e5029249588fd3f1a27b399a8d&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=5cef7e0c81c54474b23931f34295ad6e => generated 259 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:25.568453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaf3531-45e7-439f-ba56-0dc9db35b3f3 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:25.624339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1ed381-de3a-44f8-8504-13bd4d26bb79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:25.624910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ed381-de3a-44f8-8504-13bd4d26bb79 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:25.625193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ed381-de3a-44f8-8504-13bd4d26bb79 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:25.625463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ed381-de3a-44f8-8504-13bd4d26bb79 None admin] PATH_INFO: `/v3/projects/90b8410d9d4946d5ac6094702e13a192` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:25.626732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1ed381-de3a-44f8-8504-13bd4d26bb79 None admin] RBAC: Authorizing `identity:get_project(project_id=90b8410d9d4946d5ac6094702e13a192)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:25.628262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1ed381-de3a-44f8-8504-13bd4d26bb79 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:25.630367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 69/136] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:25 2022] GET /identity/v3/projects/90b8410d9d4946d5ac6094702e13a192 => generated 312 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:25.636000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:25.777285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b493bed-306f-414a-8be0-51ccf7fe7159 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:25.790558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:25.813282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab21bdbc-e091-4256-8b99-d1d2ad43ac3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:25.814037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab21bdbc-e091-4256-8b99-d1d2ad43ac3e None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:25.814320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab21bdbc-e091-4256-8b99-d1d2ad43ac3e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:25.814592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab21bdbc-e091-4256-8b99-d1d2ad43ac3e None admin] PATH_INFO: `/v3/projects/90b8410d9d4946d5ac6094702e13a192/users/38e7ae862c514700ac27c1f36f7caf25/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:25.816676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab21bdbc-e091-4256-8b99-d1d2ad43ac3e None admin] RBAC: Authorizing `identity:create_grant(project_id=90b8410d9d4946d5ac6094702e13a192, user_id=38e7ae862c514700ac27c1f36f7caf25, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:25.818190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab21bdbc-e091-4256-8b99-d1d2ad43ac3e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:25.839327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 68/137] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:25 2022] PUT /identity/v3/projects/90b8410d9d4946d5ac6094702e13a192/users/38e7ae862c514700ac27c1f36f7caf25/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:26.957939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0825fa7-011f-4f2b-b660-30798ad8814a None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:26.958560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0825fa7-011f-4f2b-b660-30798ad8814a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:26.958560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0825fa7-011f-4f2b-b660-30798ad8814a None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:26.959847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 70/138] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:26.965081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29215471-fbe3-4da5-867d-1b4c4debb391 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:26.965499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29215471-fbe3-4da5-867d-1b4c4debb391 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:26.965671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29215471-fbe3-4da5-867d-1b4c4debb391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:26.984800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-29215471-fbe3-4da5-867d-1b4c4debb391 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:26.986037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-29215471-fbe3-4da5-867d-1b4c4debb391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:27.057868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 69/139] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:27.349868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0825fa7-011f-4f2b-b660-30798ad8814a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:27.410508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:27.468506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0825fa7-011f-4f2b-b660-30798ad8814a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:27.510839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a00b69d-fa3b-4512-873d-56a742332893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:27.511483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a00b69d-fa3b-4512-873d-56a742332893 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:27.511756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a00b69d-fa3b-4512-873d-56a742332893 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:27.512021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a00b69d-fa3b-4512-873d-56a742332893 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:27.524839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a00b69d-fa3b-4512-873d-56a742332893 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-6a00b69d-fa3b-4512-873d-56a742332893 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:27.530276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:27.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:27.533514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 71/140] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:27.540278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29215471-fbe3-4da5-867d-1b4c4debb391 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:27.600432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3dc4de-3542-49a1-8103-e3e3328dd8f1 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:27.600758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3dc4de-3542-49a1-8103-e3e3328dd8f1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:27.600963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3dc4de-3542-49a1-8103-e3e3328dd8f1 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:27.602450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 72/141] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:27.608332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603ca2a-a90f-47d3-ac50-18dacb29104b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:27.608607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603ca2a-a90f-47d3-ac50-18dacb29104b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:27.608875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603ca2a-a90f-47d3-ac50-18dacb29104b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:27.630493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d603ca2a-a90f-47d3-ac50-18dacb29104b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:27.631718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d603ca2a-a90f-47d3-ac50-18dacb29104b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:27.695538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29215471-fbe3-4da5-867d-1b4c4debb391 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:27.698106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:27.736798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 73/142] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:27.751183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:27.751801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:27.752079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:27.752352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:27.753022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:27.753370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:27.754798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:27.763569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 70/143] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:27.770771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603ca2a-a90f-47d3-ac50-18dacb29104b None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:27.802586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:27.898544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603ca2a-a90f-47d3-ac50-18dacb29104b None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:27.944053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:27.944690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:27.944965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:27.945291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] PATH_INFO: `/v3/users/f6cec6cd063d4a0684ad9f283f12b406` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:27.946465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] RBAC: Authorizing `identity:get_user(user_id=f6cec6cd063d4a0684ad9f283f12b406)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:27.947921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:27.949716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 74/144] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:27 2022] GET /identity/v3/users/f6cec6cd063d4a0684ad9f283f12b406 => generated 247 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:27.955628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:28.067246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977f6ab-4834-4f99-9198-4a1461ed11f9 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:28.102107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:28.102710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.103058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.103332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.136015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:28.175038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:28.176528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:28.277402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 71/145] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:28.283359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:28.293305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:28.411339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:28.451601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142efa16-926e-4632-ba31-cc0f1c43770e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:28.454188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:28.462290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8d27ac-62b2-4b94-b57c-ad21d97ecd8a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:28.490408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:28.491100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.491381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.491666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.505128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:28.510660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:28.511600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:28.513331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 72/146] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:28.518343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:28.553990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:28.554638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.554917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.555215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.556343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:28.557786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:28.559761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 75/147] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:28 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:28.579818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:28.634221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86db9475-c21e-411a-ab51-3e1c71868ea1 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:28.692843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:28.706571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11247292-b859-4b19-9c5c-dd890cc21db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:28.707274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11247292-b859-4b19-9c5c-dd890cc21db3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.707547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11247292-b859-4b19-9c5c-dd890cc21db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.707817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11247292-b859-4b19-9c5c-dd890cc21db3 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.708505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11247292-b859-4b19-9c5c-dd890cc21db3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:28.708807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11247292-b859-4b19-9c5c-dd890cc21db3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:28.710285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11247292-b859-4b19-9c5c-dd890cc21db3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:28.721499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 73/148] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:28.722847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de28500-19fc-4e85-9762-eda02ccb7797 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.723160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de28500-19fc-4e85-9762-eda02ccb7797 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.723429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de28500-19fc-4e85-9762-eda02ccb7797 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.729703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 74/149] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:28 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:28.730639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de28500-19fc-4e85-9762-eda02ccb7797 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:28.841022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ab99f0-66cf-4aa3-b4bb-3f3f86dbcbe5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:28.879449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48157425-0dbb-4494-8049-a9fea534a915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:28.880046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48157425-0dbb-4494-8049-a9fea534a915 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.880322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48157425-0dbb-4494-8049-a9fea534a915 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.880593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48157425-0dbb-4494-8049-a9fea534a915 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.881311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48157425-0dbb-4494-8049-a9fea534a915 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=f6cec6cd063d4a0684ad9f283f12b406) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:28.881613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48157425-0dbb-4494-8049-a9fea534a915 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:28.883296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48157425-0dbb-4494-8049-a9fea534a915 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:28.890346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 76/150] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:28 2022] GET /identity/v3/role_assignments?user.id=f6cec6cd063d4a0684ad9f283f12b406&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:28.891615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21f5992-f605-4cd6-8d03-bc5f1f0eb933 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.891895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21f5992-f605-4cd6-8d03-bc5f1f0eb933 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.892163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21f5992-f605-4cd6-8d03-bc5f1f0eb933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.916240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de28500-19fc-4e85-9762-eda02ccb7797 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:28.932052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b21f5992-f605-4cd6-8d03-bc5f1f0eb933 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:28.933649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b21f5992-f605-4cd6-8d03-bc5f1f0eb933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:28.982002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:28.986213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d03f3-eae6-4365-823c-b2a74ce52544 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:28.988018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d03f3-eae6-4365-823c-b2a74ce52544 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.988475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d03f3-eae6-4365-823c-b2a74ce52544 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.989366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d03f3-eae6-4365-823c-b2a74ce52544 None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:28.991196 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d03f3-eae6-4365-823c-b2a74ce52544 None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:28.993064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d03f3-eae6-4365-823c-b2a74ce52544 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:28.996220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 75/151] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:28 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:28.997784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c37261-a42a-49a2-a437-e4527b5dd7db None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:28.998779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c37261-a42a-49a2-a437-e4527b5dd7db None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:28.999289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c37261-a42a-49a2-a437-e4527b5dd7db None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.001011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 76/152] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:29.007343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73161c07-dffe-47a0-aa82-343a927cb147 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.007890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73161c07-dffe-47a0-aa82-343a927cb147 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.008352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73161c07-dffe-47a0-aa82-343a927cb147 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.010382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 77/153] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:29.011758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73161c07-dffe-47a0-aa82-343a927cb147 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.048047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 77/154] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:29.049587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ecacc-df46-4ea2-99e6-19e6de28144b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.050285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ecacc-df46-4ea2-99e6-19e6de28144b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.051211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ecacc-df46-4ea2-99e6-19e6de28144b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.072258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f91ecacc-df46-4ea2-99e6-19e6de28144b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:29.074266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f91ecacc-df46-4ea2-99e6-19e6de28144b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:29.115541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:29.163192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 78/155] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:29.164806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e8ab1-8624-473c-b111-0df40bd3b49d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.168694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e8ab1-8624-473c-b111-0df40bd3b49d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.168694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e8ab1-8624-473c-b111-0df40bd3b49d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.177587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73161c07-dffe-47a0-aa82-343a927cb147 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:29.201151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-5f3e8ab1-8624-473c-b111-0df40bd3b49d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:29.202402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5f3e8ab1-8624-473c-b111-0df40bd3b49d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:29.239873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:29.240415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.240696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.241118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.291054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 79/156] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:29.292373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c52139-9120-4355-902a-a3e70a56004a None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.292839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c52139-9120-4355-902a-a3e70a56004a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.293162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c52139-9120-4355-902a-a3e70a56004a None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.294415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 80/157] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:29.300181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867b376c-96a1-48e5-b32a-38529d2ea067 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.300462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867b376c-96a1-48e5-b32a-38529d2ea067 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.300737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867b376c-96a1-48e5-b32a-38529d2ea067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.321501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-867b376c-96a1-48e5-b32a-38529d2ea067 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:29.322692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-867b376c-96a1-48e5-b32a-38529d2ea067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:29.325054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:29.327328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:29.352088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:29.404539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 81/158] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:29.405932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867b376c-96a1-48e5-b32a-38529d2ea067 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.422691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:29.489009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 78/159] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:29.494456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.543386 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867b376c-96a1-48e5-b32a-38529d2ea067 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:29.582732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:29.583743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.584209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.584788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.597298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:29.602702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:29.603761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:29.604793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:29.606866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 82/160] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:29.607949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.614434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6f45c6-d298-496b-873d-cd8d99bce386 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:29.624543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:29.662196 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:29.662879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.663188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.663465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.664872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:29.666324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:29.668797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 79/161] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:29.671460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.699570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:29.722302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec3b87-3156-4000-a9b1-030dc989a648 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:29.757160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:29.757697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.757974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.758242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.768735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:29.773476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:29.774546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 83/162] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:29.775810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.781355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e776c5-01e6-408a-b934-4ec2fa99f158 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:29.814822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:29.815458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.815731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.815998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.816691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:29.816992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:29.818356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:29.826743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 80/163] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:29.827533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.845418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:29.875457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef70ee4a-7a7c-4f7a-b98a-3f0a744d98ab None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:29.909362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:29.909994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.910270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.910540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.911133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:29.911672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:29.913111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:29.914675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 84/164] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:29.919495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:29.928891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce61fcc5-bcd0-49cd-a31c-f3013f7df83d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:29.962937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:29.963552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:29.963819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:29.964097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:29.964809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=38e7ae862c514700ac27c1f36f7caf25) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:29.965106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:29.966536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:29.972704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 81/165] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/role_assignments?user.id=38e7ae862c514700ac27c1f36f7caf25&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:29.988353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.020578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4cbaf-449b-48d9-a35b-81b4fde2bf7a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.059465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.060532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.061276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.061555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.070198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.073862 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:30.079186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:30.080195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:30.081004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:30.088968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 85/166] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:30.088968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.108431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a613c271-dec2-45f5-ba8d-911fe34e46ab None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.150211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.150870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.151172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.151388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.152078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] RBAC: Adding query filter params (name=reader) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:30.152344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.153749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.165035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 82/167] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:30.167087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.180647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.211792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674a7297-70ac-4edb-a21f-f9d1ea9808b5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.252331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.252952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.253225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.253493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.254610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.256095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.257665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 86/168] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:30.258408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.289381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7024dcf6-71cd-41d0-b8d4-2142db1b43ae None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.323324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.325167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.325770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.326046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.326354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.327035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.328498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.354541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 83/169] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:18:30 2022] POST /identity/v3/users => generated 283 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:30.355653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.366039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d92cd7-705e-476b-af0d-7120369db4fb None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.411584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.412212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.412491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.412765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.413586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] RBAC: Adding query filter params (name=member) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:30.413788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.415235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.425875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 87/170] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:30.426485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.435341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.482312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04001a9-e90e-4cd1-91c4-53c555c648a8 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.521425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.522073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.522347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.522614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] PATH_INFO: `/v3/users/70e4d1e5029249588fd3f1a27b399a8d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.523807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] RBAC: Authorizing `identity:get_user(user_id=70e4d1e5029249588fd3f1a27b399a8d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.525304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.526898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 84/171] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/users/70e4d1e5029249588fd3f1a27b399a8d => generated 289 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:30.527691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.548587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02a26de-43a9-4084-a8c7-0dc38307645d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.583713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.593146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.593762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.594042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.594318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.659807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f22ab2-1556-42de-b99a-da7d6f557ef6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.682901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.684748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.700279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.700916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.701350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.701626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] PATH_INFO: `/v3/users/38e7ae862c514700ac27c1f36f7caf25` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.702766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] RBAC: Authorizing `identity:get_user(user_id=38e7ae862c514700ac27c1f36f7caf25)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.704276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.705964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 85/172] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/users/38e7ae862c514700ac27c1f36f7caf25 => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:30.706734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.716326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.817839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 88/173] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:30.819956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.838680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e3c3e6-5ef6-42cf-a2d7-210346b710e3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.855897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.885963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.886684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.887051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.887391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:30.890373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:30.892058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:30.893950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 86/174] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:30.895987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:30.946585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39ca650-7322-4204-9a76-a8970e7331c5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:30.983319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:30.995234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:30.996351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:30.996844 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:30.997155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:31.019432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09065961-9631-46ba-99ec-5a7afa608a6a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:31.066025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:31.067009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:31.067736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:31.068269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:31.069756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:31.071962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:31.073929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 87/175] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:31.075344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:31.077687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:31.079299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:31.136384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:31.257552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2719769-f0c9-428d-a7e1-7c6d2244f757 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:31.284292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 89/176] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:31.285529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:31.320857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:31.321445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:31.321720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:31.321987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/70e4d1e5029249588fd3f1a27b399a8d/roles/5cef7e0c81c54474b23931f34295ad6e` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:31.325149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] RBAC: Authorizing `identity:create_grant(project_id=55dd5dfdb0894fdfad4138219ed4e9e8, user_id=70e4d1e5029249588fd3f1a27b399a8d, role_id=5cef7e0c81c54474b23931f34295ad6e)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:31.328035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:31.349253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:31.355497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 88/177] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:31 2022] PUT /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/users/70e4d1e5029249588fd3f1a27b399a8d/roles/5cef7e0c81c54474b23931f34295ad6e => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:31.356882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:31.485013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cd1c80-b9bf-4b9e-9b0a-972dc2ac7278 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:31.523356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:31.528810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:31.529757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:31.530087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:31.530692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:31.531733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] RBAC: Adding query filter params (role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=38e7ae862c514700ac27c1f36f7caf25) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:31.532095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:31.533945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:31.542596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 90/178] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:31 2022] GET /identity/v3/role_assignments?user.id=38e7ae862c514700ac27c1f36f7caf25&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 608 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:31.565061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f70962-dc93-4aca-b55f-c5ada8c77d09 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:31.619913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff549607-0b67-49df-a732-dc540ff926e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:31.620792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff549607-0b67-49df-a732-dc540ff926e2 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:31.621219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff549607-0b67-49df-a732-dc540ff926e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:31.621626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff549607-0b67-49df-a732-dc540ff926e2 None admin] PATH_INFO: `/v3/projects/90b8410d9d4946d5ac6094702e13a192` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:31.627957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff549607-0b67-49df-a732-dc540ff926e2 None admin] RBAC: Authorizing `identity:get_project(project_id=90b8410d9d4946d5ac6094702e13a192)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:31.629698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff549607-0b67-49df-a732-dc540ff926e2 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:31.631661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 89/179] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:31 2022] GET /identity/v3/projects/90b8410d9d4946d5ac6094702e13a192 => generated 312 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:31.638008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:31.785439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6517070-4118-48af-8622-b7aa01c941b0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:31.831201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6b03ad-0a79-4258-a256-9b75ccf1ad61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:31.831864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b03ad-0a79-4258-a256-9b75ccf1ad61 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:31.832142 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b03ad-0a79-4258-a256-9b75ccf1ad61 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:31.832451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b03ad-0a79-4258-a256-9b75ccf1ad61 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:31.833187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6b03ad-0a79-4258-a256-9b75ccf1ad61 None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=90b8410d9d4946d5ac6094702e13a192, user.id=38e7ae862c514700ac27c1f36f7caf25) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:31.833490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6b03ad-0a79-4258-a256-9b75ccf1ad61 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:31.834916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6b03ad-0a79-4258-a256-9b75ccf1ad61 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:31.844560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 91/180] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:31 2022] GET /identity/v3/role_assignments?user.id=38e7ae862c514700ac27c1f36f7caf25&scope.project.id=90b8410d9d4946d5ac6094702e13a192&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:33.075550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d252b0-978d-4b35-bd9f-d34345f84a32 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.076160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d252b0-978d-4b35-bd9f-d34345f84a32 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.076243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d252b0-978d-4b35-bd9f-d34345f84a32 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.077739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 90/181] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:33.083074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a47961-d6bb-4ddc-8d3f-91379413cc30 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.083478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a47961-d6bb-4ddc-8d3f-91379413cc30 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.083642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a47961-d6bb-4ddc-8d3f-91379413cc30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.102495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c4a47961-d6bb-4ddc-8d3f-91379413cc30 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:33.103749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c4a47961-d6bb-4ddc-8d3f-91379413cc30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:33.167465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:33.188589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 92/182] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:33.272397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d318a777-711a-4d6e-858a-b4d1e25e6364 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.273029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d318a777-711a-4d6e-858a-b4d1e25e6364 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.273501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d318a777-711a-4d6e-858a-b4d1e25e6364 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.275146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 91/183] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:33.280660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204e18a-59a1-4456-81ec-c79fde6c6145 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.281179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204e18a-59a1-4456-81ec-c79fde6c6145 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.281670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204e18a-59a1-4456-81ec-c79fde6c6145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.300470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d204e18a-59a1-4456-81ec-c79fde6c6145 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:33.301994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d204e18a-59a1-4456-81ec-c79fde6c6145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:33.331949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b601f0-a122-4013-aa3d-76d85a3aa59b None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.332515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b601f0-a122-4013-aa3d-76d85a3aa59b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.332981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b601f0-a122-4013-aa3d-76d85a3aa59b None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.334510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 92/184] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:33.340518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1601b67-22a3-4104-b2a2-9f7123c5894a None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.341070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1601b67-22a3-4104-b2a2-9f7123c5894a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.341542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1601b67-22a3-4104-b2a2-9f7123c5894a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.365603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c1601b67-22a3-4104-b2a2-9f7123c5894a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:33.367699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c1601b67-22a3-4104-b2a2-9f7123c5894a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:33.373614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 93/185] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:33.433210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:33.442084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 93/186] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:33.471067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204e18a-59a1-4456-81ec-c79fde6c6145 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:33.581181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204e18a-59a1-4456-81ec-c79fde6c6145 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:33.603376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:33.622421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:33.623800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.624085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.624360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.635287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:33.640669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:33.641537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:33.642792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 94/187] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:33.647803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1601b67-22a3-4104-b2a2-9f7123c5894a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:33.649879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:33.762316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3882918-1169-4ef2-ab49-58c7b87975ae None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:33.773195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1601b67-22a3-4104-b2a2-9f7123c5894a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:33.807224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:33.807828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.808107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.808368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.809029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:33.809330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:33.810096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:33.811207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.811374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:33.811514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.811733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.825152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 95/188] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:33.825152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:33.825403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:33.827381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:33.828241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:33.829315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 94/189] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:33.831858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:33.873365 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:33.903765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:33.939892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a85d4-a5a3-4ee8-a59c-695407f7ee29 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:33.949655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eb7300-a104-401b-ba97-adf1cfee18a4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:33.976372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:33.976996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.977274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.977704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:33.978647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:33.980113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:33.981876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 96/190] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:33.983483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:33.988312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:33.989206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:33.989714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:33.990188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] PATH_INFO: `/v3/users/417ece27f2f74031b865d40b6176b1b5` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.008741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] RBAC: Authorizing `identity:get_user(user_id=417ece27f2f74031b865d40b6176b1b5)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.010512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.012454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 95/191] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:33 2022] GET /identity/v3/users/417ece27f2f74031b865d40b6176b1b5 => generated 283 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:34.013426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.100796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4e103-a9ce-4792-867d-1a8b5db7fc9c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.110035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.127468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a95df50-2e8c-4c03-8f09-3396042e2f6c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.137848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.139299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.139587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.139859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.140520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:34.140819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.142181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.143322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.150672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 97/192] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:34.151637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.167705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f59148-c3dd-47eb-8e11-03e2d15166a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.168308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f59148-c3dd-47eb-8e11-03e2d15166a0 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.168586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f59148-c3dd-47eb-8e11-03e2d15166a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.168856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f59148-c3dd-47eb-8e11-03e2d15166a0 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.169509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f59148-c3dd-47eb-8e11-03e2d15166a0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.171531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f59148-c3dd-47eb-8e11-03e2d15166a0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.202991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 96/193] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:18:34 2022] POST /identity/v3/users => generated 297 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:34.202991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3b79ef-eed0-4b9b-bd7b-bd71c21b1de7 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.202991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3b79ef-eed0-4b9b-bd7b-bd71c21b1de7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.202991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3b79ef-eed0-4b9b-bd7b-bd71c21b1de7 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.202991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 97/194] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:34.202991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3b79ef-eed0-4b9b-bd7b-bd71c21b1de7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.277606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f10882a-a2b8-4382-b89c-90fb86653e67 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.319602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.320217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.320492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.321023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.324794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3b79ef-eed0-4b9b-bd7b-bd71c21b1de7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.361336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5d9533-92e9-4513-ae0a-d11f21baee2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.362847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d9533-92e9-4513-ae0a-d11f21baee2c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.362847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d9533-92e9-4513-ae0a-d11f21baee2c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.362847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5d9533-92e9-4513-ae0a-d11f21baee2c None admin] PATH_INFO: `/v3/users/417ece27f2f74031b865d40b6176b1b5` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.363916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d9533-92e9-4513-ae0a-d11f21baee2c None admin] RBAC: Authorizing `identity:get_user(user_id=417ece27f2f74031b865d40b6176b1b5)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.364843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.365387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5d9533-92e9-4513-ae0a-d11f21baee2c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.367373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 98/195] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:34 2022] GET /identity/v3/users/417ece27f2f74031b865d40b6176b1b5 => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:34.368620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b260850-5218-4ce9-a62a-c5f941d1e615 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.369022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b260850-5218-4ce9-a62a-c5f941d1e615 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.369292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b260850-5218-4ce9-a62a-c5f941d1e615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.388604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5b260850-5218-4ce9-a62a-c5f941d1e615 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:34.391053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5b260850-5218-4ce9-a62a-c5f941d1e615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:34.406382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.407901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.422994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.480298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 99/196] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:34.481861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b260850-5218-4ce9-a62a-c5f941d1e615 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.535886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 98/197] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:34.541454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.587961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b260850-5218-4ce9-a62a-c5f941d1e615 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.624577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.625191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.625463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.625726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.632161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.666685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.672600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f94d1e-84f2-40d5-8cc6-57cd4de1193d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.704050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.705635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.710863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.711549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.711890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.712169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.713325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.714774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.717613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 99/198] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:34 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:34.723446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.815850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 100/199] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:34.816959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.838649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1060c5-1cef-4edc-bd94-3a8701c28de3 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.877558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.878164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.878442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.878716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.879650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=417ece27f2f74031b865d40b6176b1b5) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:34.879973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:34.881408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:34.887713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 100/200] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:34 2022] GET /identity/v3/role_assignments?user.id=417ece27f2f74031b865d40b6176b1b5&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:34.889097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:34.890608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.901734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:34.930806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1768328a-9d06-49d1-b517-1cbf9c28c058 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:34.970041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:34.970781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:34.971184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:34.971453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:34.984041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:34.989410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:34.991776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:34.993803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 101/201] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:35.001885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:35.027753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7573d202-224e-40fb-94ae-6c6cd14c0919 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:35.096891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:35.097537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:35.097814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:35.098090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:35.099468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:35.102330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:35.103034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 101/202] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:34 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:35.109695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:35.211463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:35.217802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73572fdc-78c7-475d-843f-48584d3cc264 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:35.259978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a92818-48a8-455c-a5ee-28786ac227b8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:35.273348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:35.289391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:35.290003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:35.290284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:35.290553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:35.291253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] RBAC: Adding query filter params (name=reader) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:35.291552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:35.294769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:35.307480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 102/203] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:35 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:35.314900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:35.327712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:35.328323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:35.328600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:35.328926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:35.329711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] RBAC: Adding query filter params (role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=417ece27f2f74031b865d40b6176b1b5) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:35.330022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:35.331487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:35.344207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 102/204] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:35 2022] GET /identity/v3/role_assignments?user.id=417ece27f2f74031b865d40b6176b1b5&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 259 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:35.477941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24335ce-ea0a-4130-8aec-f4a4a1aabefa None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:35.524414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:35.525151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:35.525430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:35.525701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] PATH_INFO: `/v3/users/70e4d1e5029249588fd3f1a27b399a8d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:35.531909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] RBAC: Authorizing `identity:get_user(user_id=70e4d1e5029249588fd3f1a27b399a8d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:35.533424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:35.535231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 103/205] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:35 2022] GET /identity/v3/users/70e4d1e5029249588fd3f1a27b399a8d => generated 289 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:35.540958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:35.677021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3b6932-7bc3-4b57-91d8-59bd8c4a05ef None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:35.725029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:35.725645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:35.725923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:35.726195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:35.729471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:35.820701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:35.822158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:35.932208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 103/206] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:35.936395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:36.028632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:36.082047 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f5777-e9af-4c03-b6ec-a3adb92a24be None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:36.121437 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a895020-7d4a-4be0-94fd-c26ef6eea9d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:36.122073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a895020-7d4a-4be0-94fd-c26ef6eea9d5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:36.122358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a895020-7d4a-4be0-94fd-c26ef6eea9d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:36.122630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a895020-7d4a-4be0-94fd-c26ef6eea9d5 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:36.124043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a895020-7d4a-4be0-94fd-c26ef6eea9d5 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:36.125619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a895020-7d4a-4be0-94fd-c26ef6eea9d5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:36.127611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 104/207] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:35 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:36.134917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:36.199345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:36.283981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269d000d-c009-4c4d-a0ff-12d690ef3bab None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:36.334502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc535352-4004-48cd-82df-85850598b81f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:36.335135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc535352-4004-48cd-82df-85850598b81f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:36.335407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc535352-4004-48cd-82df-85850598b81f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:36.335673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc535352-4004-48cd-82df-85850598b81f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:36.336392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc535352-4004-48cd-82df-85850598b81f None admin] RBAC: Adding query filter params (role.id=5cef7e0c81c54474b23931f34295ad6e, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8, user.id=70e4d1e5029249588fd3f1a27b399a8d) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:36.336689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc535352-4004-48cd-82df-85850598b81f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:36.338095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc535352-4004-48cd-82df-85850598b81f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:36.344209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 104/208] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:36 2022] GET /identity/v3/role_assignments?user.id=70e4d1e5029249588fd3f1a27b399a8d&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=5cef7e0c81c54474b23931f34295ad6e => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:36.944859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e996da2-fd31-4bd5-8612-b0423320e097 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:36.945404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e996da2-fd31-4bd5-8612-b0423320e097 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:36.945404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e996da2-fd31-4bd5-8612-b0423320e097 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:36.947050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 105/209] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:36.954429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7392f431-fb24-445c-b51e-575afec13fd9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:36.954715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7392f431-fb24-445c-b51e-575afec13fd9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:36.955009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7392f431-fb24-445c-b51e-575afec13fd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:36.981935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7392f431-fb24-445c-b51e-575afec13fd9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:36.983597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7392f431-fb24-445c-b51e-575afec13fd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:37.065824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 105/210] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:37.130452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52c29c4-d5ff-45bd-8968-12ceb7432724 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.130755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52c29c4-d5ff-45bd-8968-12ceb7432724 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.131028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52c29c4-d5ff-45bd-8968-12ceb7432724 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.132402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 106/211] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:37.137633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3b646-09bf-4210-be6a-fe4c47a2a0ea None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.138072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3b646-09bf-4210-be6a-fe4c47a2a0ea None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.138237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3b646-09bf-4210-be6a-fe4c47a2a0ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.157528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-1cc3b646-09bf-4210-be6a-fe4c47a2a0ea None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:37.158720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1cc3b646-09bf-4210-be6a-fe4c47a2a0ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:37.189972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:37.244984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 106/212] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:37.305390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54eead7-e9ca-44a9-881f-a669e52959e7 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.305742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54eead7-e9ca-44a9-881f-a669e52959e7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.305933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54eead7-e9ca-44a9-881f-a669e52959e7 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.307556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 107/213] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:37.313288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771ef64f-0d58-468e-b50e-44f509e00a39 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.313573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771ef64f-0d58-468e-b50e-44f509e00a39 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.313842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771ef64f-0d58-468e-b50e-44f509e00a39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.333493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-771ef64f-0d58-468e-b50e-44f509e00a39 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:37.334664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-771ef64f-0d58-468e-b50e-44f509e00a39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:37.354450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54eead7-e9ca-44a9-881f-a669e52959e7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:37.408243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 107/214] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:37.469029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54eead7-e9ca-44a9-881f-a669e52959e7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:37.494951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:37.505796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:37.506467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.506743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.507038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.508137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:37.509630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:37.511497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 108/215] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:37.517528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771ef64f-0d58-468e-b50e-44f509e00a39 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:37.531206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:37.632978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:37.649453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771ef64f-0d58-468e-b50e-44f509e00a39 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:37.678774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b15e67-2c22-4904-8a4b-c6d019b75e8a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:37.695378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:37.696374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.696801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.697073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.697738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:37.699627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:37.720782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:37.721413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.721689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.721954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.723220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 108/216] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:18:37 2022] POST /identity/v3/users => generated 297 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:37.723894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:37.738797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:37.744494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:37.745565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:37.746413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 109/217] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:18:37.753218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:37.838268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:37.869617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f8875-2bb4-4d6e-b428-387dc70e6fe4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:37.906790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34873730-4e48-4fc2-841b-fdd22aafbf9f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:37.920996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:37.921981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.922527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.923030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.934659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:37.941157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:37.942087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:37.942887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:37.945190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 109/218] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:37.948663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:37.948663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:37.949180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:37.949180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:37.950099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:37.950399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:37.951743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:37.952360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:37.956268 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:37.962491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 110/219] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:37.971397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:38.084182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036f9ab9-9a7c-4bde-9825-6b86069ffe2f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.098479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3f4a2-b87f-4aef-ae77-c96243c3d783 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.121132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:38.128902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:38.129765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.130392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.130392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.131117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:38.131413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:38.132914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:38.143471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:38.144459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 110/220] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:38.145217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.145495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.145767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] PATH_INFO: `/v3/users/07320404c7be4223a89211cb9888a17e` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.151105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:38.166937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] RBAC: Authorizing `identity:get_user(user_id=07320404c7be4223a89211cb9888a17e)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:38.168545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:38.171322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 111/221] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:37 2022] GET /identity/v3/users/07320404c7be4223a89211cb9888a17e => generated 297 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:38.176568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:38.243367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:38.306314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2c766-4a76-4b94-82fc-b51b23ad5e37 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.345161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769e47aa-61db-4200-a07e-80e5c5166723 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.347988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fa0ff5-6eb9-4bf1-a4db-e39939fcc758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:38.348613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fa0ff5-6eb9-4bf1-a4db-e39939fcc758 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.348935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fa0ff5-6eb9-4bf1-a4db-e39939fcc758 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.349212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fa0ff5-6eb9-4bf1-a4db-e39939fcc758 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.424893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fa0ff5-6eb9-4bf1-a4db-e39939fcc758 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:38.426480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fa0ff5-6eb9-4bf1-a4db-e39939fcc758 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:38.427021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:38.427855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.428305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.428651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:38.428776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] PATH_INFO: `/v3/users/417ece27f2f74031b865d40b6176b1b5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.435863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] RBAC: Authorizing `identity:get_user(user_id=417ece27f2f74031b865d40b6176b1b5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:38.437538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:38.439606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 111/222] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:38 2022] GET /identity/v3/users/417ece27f2f74031b865d40b6176b1b5 => generated 283 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:38.445216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:38.552297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 112/223] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 06 04:18:38.553751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6272d7f5-28d4-4c19-9dea-487f806f8715 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.554126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6272d7f5-28d4-4c19-9dea-487f806f8715 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.554448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6272d7f5-28d4-4c19-9dea-487f806f8715 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.556402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 113/224] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:38.561704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6272d7f5-28d4-4c19-9dea-487f806f8715 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:38.578058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6192ba6-f0fc-4dc7-a94e-76dba909ddee None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.616978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b953e7e-57e2-4e82-9373-50993ad7d167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:38.618478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b953e7e-57e2-4e82-9373-50993ad7d167 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.618951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b953e7e-57e2-4e82-9373-50993ad7d167 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.619692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b953e7e-57e2-4e82-9373-50993ad7d167 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.621172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b953e7e-57e2-4e82-9373-50993ad7d167 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:38.623700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b953e7e-57e2-4e82-9373-50993ad7d167 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:38.625975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 112/225] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:38 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:38.627490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920fd40f-3c3b-4f25-bdab-36ed9c58f184 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.628196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920fd40f-3c3b-4f25-bdab-36ed9c58f184 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.629054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920fd40f-3c3b-4f25-bdab-36ed9c58f184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.636641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:38.649960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:38.655753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-920fd40f-3c3b-4f25-bdab-36ed9c58f184 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:38.657948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-920fd40f-3c3b-4f25-bdab-36ed9c58f184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:38.680675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6272d7f5-28d4-4c19-9dea-487f806f8715 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.724392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:38.725025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:38.725297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:38.725561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:38.727114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:38.728926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:38.731317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 114/226] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:38 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:38.732294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:38.743821 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 113/227] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:38.751538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920fd40f-3c3b-4f25-bdab-36ed9c58f184 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:38.928998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca837ae0-0751-4379-ab76-1a7965fa3d96 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.947385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920fd40f-3c3b-4f25-bdab-36ed9c58f184 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:38.993493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:38.999661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:39.000356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:39.000635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:39.000909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47/users/417ece27f2f74031b865d40b6176b1b5/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:39.005073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] RBAC: Authorizing `identity:create_grant(project_id=5698e9901f634a3695852e6b8b460d47, user_id=417ece27f2f74031b865d40b6176b1b5, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:39.006584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:39.007753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:39.008379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:39.008657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:39.009185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:39.010385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=07320404c7be4223a89211cb9888a17e) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:39.010825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:39.012302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:39.028803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 114/228] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:38 2022] GET /identity/v3/role_assignments?user.id=07320404c7be4223a89211cb9888a17e&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 259 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:39.031218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 115/229] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:38 2022] PUT /identity/v3/projects/5698e9901f634a3695852e6b8b460d47/users/417ece27f2f74031b865d40b6176b1b5/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:39.071801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:39.194386 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f9de01-9a54-4a4e-97e7-818f576e7a54 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:39.243665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:39.244323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:39.244603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:39.244875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:39.261291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:39.267771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:39.268751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:39.269542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 115/230] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:39.277245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:39.283974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:39.424352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c617e-c4fb-4edc-ae98-06b95920be31 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:39.463555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:39.464086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:39.464363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:39.464636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:39.465318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:39.465623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:39.467318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:39.476307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 116/231] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:39.482081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:39.608762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:39.712149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5741b17-46d5-441c-8104-8f75a1c24066 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:39.766468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:39.767038 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:39.767315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:39.767587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] PATH_INFO: `/v3/users/417ece27f2f74031b865d40b6176b1b5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:39.768708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] RBAC: Authorizing `identity:get_user(user_id=417ece27f2f74031b865d40b6176b1b5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:39.770137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:39.776186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 116/232] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:39 2022] GET /identity/v3/users/417ece27f2f74031b865d40b6176b1b5 => generated 283 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:39.780473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:39.907004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:39.956090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1108f7-a9be-4f59-b607-bea7db4b0492 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:40.006859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea8b2f5-63e5-45ff-8b57-c238ff023a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:40.007522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8b2f5-63e5-45ff-8b57-c238ff023a76 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:40.007774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8b2f5-63e5-45ff-8b57-c238ff023a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:40.008052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8b2f5-63e5-45ff-8b57-c238ff023a76 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:40.009183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea8b2f5-63e5-45ff-8b57-c238ff023a76 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:40.010647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea8b2f5-63e5-45ff-8b57-c238ff023a76 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:40.012636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 117/233] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:39 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:40.019068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:40.172630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ea0916-c847-4631-a138-2d76a3a90171 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:40.195854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:40.219350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aacd195-e671-46d5-8c21-97657c84082b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:40.220031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aacd195-e671-46d5-8c21-97657c84082b None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:40.220308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aacd195-e671-46d5-8c21-97657c84082b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:40.220582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aacd195-e671-46d5-8c21-97657c84082b None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47/users/417ece27f2f74031b865d40b6176b1b5/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:40.222881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aacd195-e671-46d5-8c21-97657c84082b None admin] RBAC: Authorizing `identity:create_grant(project_id=5698e9901f634a3695852e6b8b460d47, user_id=417ece27f2f74031b865d40b6176b1b5, role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:40.225827 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aacd195-e671-46d5-8c21-97657c84082b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:40.251382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 117/234] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:40 2022] PUT /identity/v3/projects/5698e9901f634a3695852e6b8b460d47/users/417ece27f2f74031b865d40b6176b1b5/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:40.536528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20f89b-aac4-4334-8e66-16bd16e2e4e5 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:40.537019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20f89b-aac4-4334-8e66-16bd16e2e4e5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:40.537404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a20f89b-aac4-4334-8e66-16bd16e2e4e5 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:40.539125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 118/235] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:40.544115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bb9357-24e7-45b8-a9e8-05d8863a2711 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:40.544429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bb9357-24e7-45b8-a9e8-05d8863a2711 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:40.544707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bb9357-24e7-45b8-a9e8-05d8863a2711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:40.572394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f6bb9357-24e7-45b8-a9e8-05d8863a2711 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:40.573557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f6bb9357-24e7-45b8-a9e8-05d8863a2711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:40.625623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50af572d-08e8-4c31-942e-3f6dd9a8988a None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:40.625955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50af572d-08e8-4c31-942e-3f6dd9a8988a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:40.626170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50af572d-08e8-4c31-942e-3f6dd9a8988a None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:40.627770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 119/236] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:40.633092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb355b-3cf9-41e3-9681-25682d90b96b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:40.633608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb355b-3cf9-41e3-9681-25682d90b96b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:40.634108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb355b-3cf9-41e3-9681-25682d90b96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:40.653629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-80fb355b-3cf9-41e3-9681-25682d90b96b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:40.657240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-80fb355b-3cf9-41e3-9681-25682d90b96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:40.666599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 118/237] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:40.780073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 120/238] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:41.058474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bb9357-24e7-45b8-a9e8-05d8863a2711 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:41.064028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb355b-3cf9-41e3-9681-25682d90b96b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:41.128854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:41.190648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb355b-3cf9-41e3-9681-25682d90b96b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:41.237000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bb9357-24e7-45b8-a9e8-05d8863a2711 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:41.265715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:41.267544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:41.269148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.269754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.271431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.293947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:41.300805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:41.301735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:41.302538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:41.307374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 121/239] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:41.308730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:41.309536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.314876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.315407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.316858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:41.317902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:41.318882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:41.320991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 119/240] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:41.326686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:41.510943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c763d271-283d-44e5-8a56-1f6c0c564931 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:41.530889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce2ce98-f003-43bd-acd2-e5f048c8c6f2 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:41.573435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:41.574411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.574885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.575378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.576487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:41.578092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:41.579348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:41.608027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e894f20-2b2a-4cff-a98b-d54e88f6b3b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:41.608974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e894f20-2b2a-4cff-a98b-d54e88f6b3b7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.609464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e894f20-2b2a-4cff-a98b-d54e88f6b3b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.609956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e894f20-2b2a-4cff-a98b-d54e88f6b3b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.610344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 120/241] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:18:41 2022] POST /identity/v3/users => generated 293 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:41.611309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e894f20-2b2a-4cff-a98b-d54e88f6b3b7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:41.611854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e894f20-2b2a-4cff-a98b-d54e88f6b3b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:41.613729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e894f20-2b2a-4cff-a98b-d54e88f6b3b7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:41.625249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 122/242] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:41.631719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:41.680995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:41.750726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965b9f29-cd19-41b9-a7df-5f01938773f6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:41.756407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19adfa9-07f9-4950-9095-13151b18189d None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.756725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19adfa9-07f9-4950-9095-13151b18189d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.756950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19adfa9-07f9-4950-9095-13151b18189d None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.758586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 123/243] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:41.764718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5760b-4400-4906-a285-e9f2863dd5af None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.765140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5760b-4400-4906-a285-e9f2863dd5af None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.766043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5760b-4400-4906-a285-e9f2863dd5af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.801152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d3e5760b-4400-4906-a285-e9f2863dd5af None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:41.802421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d3e5760b-4400-4906-a285-e9f2863dd5af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:41.823782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:41.824350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.824631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.830527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] PATH_INFO: `/v3/users/191f3e755aa64b249e73ee2f700f6cee` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.853401 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] RBAC: Authorizing `identity:get_user(user_id=191f3e755aa64b249e73ee2f700f6cee)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:41.854883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:41.856753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 121/244] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:41 2022] GET /identity/v3/users/191f3e755aa64b249e73ee2f700f6cee => generated 297 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:41.862945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:41.917128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 124/245] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:41.918865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c445f3c9-07ab-4052-814d-0418d1d46341 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.920000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c445f3c9-07ab-4052-814d-0418d1d46341 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.920509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c445f3c9-07ab-4052-814d-0418d1d46341 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.921985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 125/246] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:41.927639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213086d-3c41-444c-8177-108129d64463 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:41.928156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213086d-3c41-444c-8177-108129d64463 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:41.928624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213086d-3c41-444c-8177-108129d64463 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:41.953195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5213086d-3c41-444c-8177-108129d64463 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:41.955626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5213086d-3c41-444c-8177-108129d64463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:41.991050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:42.001248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdc5d72-8b8e-496a-880e-a09505d5221c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:42.047422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 126/247] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:42.049293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:42.049956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:42.050233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:42.050495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:42.129398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:42.132153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:42.236430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5213086d-3c41-444c-8177-108129d64463 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:42.249929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:42.256596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 122/248] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:42.262658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:42.300878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:42.364516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5213086d-3c41-444c-8177-108129d64463 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:42.369515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edaea76-9cfc-4c4d-820a-94d74f9ec679 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:42.403265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:42.404378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:42.404658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:42.404929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:42.412310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:42.412951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:42.413228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:42.413500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:42.414642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:42.416126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:42.416440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:42.417903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 123/249] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:42.418655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:42.421292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:42.422199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:42.423410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 127/250] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:18:42.426115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:42.510522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:42.557197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b9031d-0d8b-4ab3-938c-e3d2cf6b0cad None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:42.569905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae0f867-3910-47f4-bf14-179cdeb894ce None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:42.573185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:42.597759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:42.598375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:42.598807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:42.599258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:42.611641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:42.617764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:42.618817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:18:42.619852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:42.621721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 124/251] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:42.622786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:42.626714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:42.628320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:42.628599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:42.628875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:42.629615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] RBAC: Adding query filter params (role.id=5cef7e0c81c54474b23931f34295ad6e, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=191f3e755aa64b249e73ee2f700f6cee) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:42.629922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:42.631411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:42.644428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 128/252] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/role_assignments?user.id=191f3e755aa64b249e73ee2f700f6cee&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=5cef7e0c81c54474b23931f34295ad6e => generated 259 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:42.646193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:42.824594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45a8b3-7217-4001-aeac-aa3bf3eb9afb None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:42.854094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb270c5-6454-4167-b597-300cea915804 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:42.871283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:42.891405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:42.892260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:42.892675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:42.893082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:42.893916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] RBAC: Adding query filter params (name=member) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:42.894354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:42.895905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:42.911311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 125/253] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:42.921809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:42.933198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:42.934076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:42.934499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:42.934947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:42.935933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:42.936423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:42.938035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:42.956419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 129/254] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:42.961907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:43.036560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:43.121246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7e6f3d-9adf-4670-99bb-6ae3092e28aa None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:43.161711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19b1eb7-1bce-4fef-aab2-1fa744d06cd9 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:43.187801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8eba0dc-4314-4f23-ba0f-122c257640bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:43.188442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eba0dc-4314-4f23-ba0f-122c257640bc None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:43.188719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eba0dc-4314-4f23-ba0f-122c257640bc None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:43.188990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eba0dc-4314-4f23-ba0f-122c257640bc None admin] PATH_INFO: `/v3/users/07320404c7be4223a89211cb9888a17e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:43.191690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eba0dc-4314-4f23-ba0f-122c257640bc None admin] RBAC: Authorizing `identity:get_user(user_id=07320404c7be4223a89211cb9888a17e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:43.193141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eba0dc-4314-4f23-ba0f-122c257640bc None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:43.194941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 126/255] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/users/07320404c7be4223a89211cb9888a17e => generated 297 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:43.196110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255eda6-fc5e-4d2e-aec7-7b18f0bbcd89 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:43.196389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255eda6-fc5e-4d2e-aec7-7b18f0bbcd89 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:43.196659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8255eda6-fc5e-4d2e-aec7-7b18f0bbcd89 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:43.197872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 127/256] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:43 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:43.201841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255eda6-fc5e-4d2e-aec7-7b18f0bbcd89 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:43.231043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bdd096-49cf-4433-9aa2-06f953e1b928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:43.231686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdd096-49cf-4433-9aa2-06f953e1b928 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:43.232014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdd096-49cf-4433-9aa2-06f953e1b928 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:43.232394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdd096-49cf-4433-9aa2-06f953e1b928 None admin] PATH_INFO: `/v3/users/417ece27f2f74031b865d40b6176b1b5` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:43.233988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bdd096-49cf-4433-9aa2-06f953e1b928 None admin] RBAC: Authorizing `identity:get_user(user_id=417ece27f2f74031b865d40b6176b1b5)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:43.235503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bdd096-49cf-4433-9aa2-06f953e1b928 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:43.237410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 130/257] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:42 2022] GET /identity/v3/users/417ece27f2f74031b865d40b6176b1b5 => generated 283 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:43.238532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177ba99-e5d4-4a15-9631-42339d1014a5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:43.238809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177ba99-e5d4-4a15-9631-42339d1014a5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:43.243393 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c177ba99-e5d4-4a15-9631-42339d1014a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:43.263474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c177ba99-e5d4-4a15-9631-42339d1014a5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:43.264790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c177ba99-e5d4-4a15-9631-42339d1014a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:43.363316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:43.383563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 131/258] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:43.387034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c177ba99-e5d4-4a15-9631-42339d1014a5 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:43.437158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:43.448806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8255eda6-fc5e-4d2e-aec7-7b18f0bbcd89 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:43.505091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:43.505720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:43.505996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:43.506269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:43.507699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:43.509132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:43.510768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 128/259] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:43 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:43.516888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:43.530638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c177ba99-e5d4-4a15-9631-42339d1014a5 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:43.580549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:43.581529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:43.581997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:43.582458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:43.639333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ff4d9b-a4f9-4f9a-a639-90a831171225 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:43.670984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:43.677278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:43.677940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:43.678218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:43.678492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47/users/07320404c7be4223a89211cb9888a17e/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:43.681365 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] RBAC: Authorizing `identity:create_grant(project_id=5698e9901f634a3695852e6b8b460d47, user_id=07320404c7be4223a89211cb9888a17e, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:43.683543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:43.711466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 129/260] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:43 2022] PUT /identity/v3/projects/5698e9901f634a3695852e6b8b460d47/users/07320404c7be4223a89211cb9888a17e/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:18:43.715956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:43.717751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:43.759788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:43.813655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:43.860870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 132/261] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:43.867027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:44.000778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a722bc84-a0ad-4803-ba4c-fa1518c181b0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:44.012086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8dfd03-c356-4e0b-b2ee-b788e5064088 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:44.047235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:44.056975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:44.057918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:44.058387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:44.058847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:44.060561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:44.062201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:44.064363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 133/262] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:43 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:44.070099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:44.071262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:44.074241 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:44.075541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:44.076104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:44.098148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:44.104815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:44.105715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:44.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:44.111193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 130/263] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:18:44.118668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:44.207313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:44.266935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828b140e-61e9-4ef8-a9f7-47b402d4fa3e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:44.336996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:44.337940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:44.338225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:44.338499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:44.339282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=417ece27f2f74031b865d40b6176b1b5) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:44.339593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:44.341561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:44.344878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dbeb03-c1c3-4d1c-b133-d0688a158070 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:44.354795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 134/264] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:44 2022] GET /identity/v3/role_assignments?user.id=417ece27f2f74031b865d40b6176b1b5&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:44.414112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:44.414792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:44.415070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:44.415344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:44.416014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:44.416405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:44.417889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:44.433605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 131/265] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:44.438996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:44.546637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:44.608322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0478a1f8-619f-4921-946a-ad51931669eb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:44.655147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:44.655810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:44.656092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:44.656363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] PATH_INFO: `/v3/users/417ece27f2f74031b865d40b6176b1b5` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:44.661825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] RBAC: Authorizing `identity:get_user(user_id=417ece27f2f74031b865d40b6176b1b5)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:44.663452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:44.665576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 135/266] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:44 2022] GET /identity/v3/users/417ece27f2f74031b865d40b6176b1b5 => generated 283 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:44.671574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:44.788153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da110d34-a890-40ec-80c6-b59efd578b4b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:44.790997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:44.825378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dba99b-f9e8-4946-9153-5ba99b885201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:44.826374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dba99b-f9e8-4946-9153-5ba99b885201 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:44.826660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dba99b-f9e8-4946-9153-5ba99b885201 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:44.826983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dba99b-f9e8-4946-9153-5ba99b885201 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:44.905188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dba99b-f9e8-4946-9153-5ba99b885201 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:44.906670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dba99b-f9e8-4946-9153-5ba99b885201 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:45.023812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 132/267] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:45.030311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:45.060283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0293a39-c83d-4da2-aaf1-92ec16cf51db None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:45.060638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0293a39-c83d-4da2-aaf1-92ec16cf51db None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:45.060909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0293a39-c83d-4da2-aaf1-92ec16cf51db None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:45.062602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 133/268] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:45.071742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff3e23d-8d59-46dd-b698-84b9037e8ec6 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:45.072103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff3e23d-8d59-46dd-b698-84b9037e8ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:45.072498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff3e23d-8d59-46dd-b698-84b9037e8ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:45.085007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:45.114276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3ff3e23d-8d59-46dd-b698-84b9037e8ec6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:45.115509 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3ff3e23d-8d59-46dd-b698-84b9037e8ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:45.208398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997a661c-2d31-4277-923b-3bc6ab60cd25 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:45.233304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 134/269] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:45.253421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:45.258908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2e30de-80da-4bbd-9af3-bb275b0ff548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:45.259608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2e30de-80da-4bbd-9af3-bb275b0ff548 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:45.259903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2e30de-80da-4bbd-9af3-bb275b0ff548 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:45.260177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2e30de-80da-4bbd-9af3-bb275b0ff548 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:45.261477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2e30de-80da-4bbd-9af3-bb275b0ff548 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:45.264736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2e30de-80da-4bbd-9af3-bb275b0ff548 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:45.266705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 136/270] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:45 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:45.276064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff3e23d-8d59-46dd-b698-84b9037e8ec6 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:45.400542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff3e23d-8d59-46dd-b698-84b9037e8ec6 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:45.437734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8932e1cc-9902-4248-9468-0eb2bb7f8bc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:45.439180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8932e1cc-9902-4248-9468-0eb2bb7f8bc5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:45.439678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8932e1cc-9902-4248-9468-0eb2bb7f8bc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:45.440179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8932e1cc-9902-4248-9468-0eb2bb7f8bc5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:45.441153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8932e1cc-9902-4248-9468-0eb2bb7f8bc5 None admin] RBAC: Adding query filter params (role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=417ece27f2f74031b865d40b6176b1b5) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:45.441674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8932e1cc-9902-4248-9468-0eb2bb7f8bc5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:45.443387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8932e1cc-9902-4248-9468-0eb2bb7f8bc5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:45.445394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:45.450277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 135/271] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:45 2022] GET /identity/v3/role_assignments?user.id=417ece27f2f74031b865d40b6176b1b5&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:45.478993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde72059-3c04-4f42-81dd-583895c438bc None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:45.479517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde72059-3c04-4f42-81dd-583895c438bc None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:45.479646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde72059-3c04-4f42-81dd-583895c438bc None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:45.481524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 137/272] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:45.487304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2d241-46e3-4dab-a761-ded1fe0a160e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:45.487600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2d241-46e3-4dab-a761-ded1fe0a160e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:45.487874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2d241-46e3-4dab-a761-ded1fe0a160e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:45.516245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f2b2d241-46e3-4dab-a761-ded1fe0a160e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:45.517495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f2b2d241-46e3-4dab-a761-ded1fe0a160e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:45.596677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 136/273] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:45.677325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde72059-3c04-4f42-81dd-583895c438bc None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:45.799536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde72059-3c04-4f42-81dd-583895c438bc None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:45.839283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:45.839921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:45.840201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:45.840475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:45.852493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:45.858186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:45.859122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:45.859970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 138/274] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:45.867521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2d241-46e3-4dab-a761-ded1fe0a160e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:45.926309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:46.011635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2d241-46e3-4dab-a761-ded1fe0a160e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:46.014599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:46.045627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:46.054886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f3cfa4-497f-4f14-bf1e-866f76274372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:46.055749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3cfa4-497f-4f14-bf1e-866f76274372 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.056167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3cfa4-497f-4f14-bf1e-866f76274372 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.056618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f3cfa4-497f-4f14-bf1e-866f76274372 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.057431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f3cfa4-497f-4f14-bf1e-866f76274372 None admin] RBAC: Adding query filter params (name=member) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:46.057731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f3cfa4-497f-4f14-bf1e-866f76274372 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:46.059118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f3cfa4-497f-4f14-bf1e-866f76274372 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:46.069335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 137/275] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:46.070444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2fa4b-899a-4acc-859b-189875976de6 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.070723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2fa4b-899a-4acc-859b-189875976de6 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.071020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b2fa4b-899a-4acc-859b-189875976de6 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.072224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 138/276] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:46 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:46.077155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2fa4b-899a-4acc-859b-189875976de6 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:46.086519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9280b121-8655-4c38-9be6-2d5314d58045 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:46.126924 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035cff87-669d-4e0a-a58b-d5cc9cadb742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:46.127707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035cff87-669d-4e0a-a58b-d5cc9cadb742 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.128011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035cff87-669d-4e0a-a58b-d5cc9cadb742 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.128564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035cff87-669d-4e0a-a58b-d5cc9cadb742 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.130123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035cff87-669d-4e0a-a58b-d5cc9cadb742 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:46.131895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035cff87-669d-4e0a-a58b-d5cc9cadb742 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:46.138618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 139/277] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:46.138618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b050c3-86f6-4dae-b2d6-55df354d58ac None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.138618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b050c3-86f6-4dae-b2d6-55df354d58ac None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.138618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b050c3-86f6-4dae-b2d6-55df354d58ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.159270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f6b050c3-86f6-4dae-b2d6-55df354d58ac None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:46.160469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f6b050c3-86f6-4dae-b2d6-55df354d58ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:46.191028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b2fa4b-899a-4acc-859b-189875976de6 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:46.233636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:46.234309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.234583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.234850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] PATH_INFO: `/v3/users/07a6cbe4a171479a9b871723789b3101` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.238525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 140/278] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:46.239958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b050c3-86f6-4dae-b2d6-55df354d58ac None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:46.249782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] RBAC: Authorizing `identity:get_user(user_id=07a6cbe4a171479a9b871723789b3101)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:46.251574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:46.253206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 139/279] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:46 2022] GET /identity/v3/users/07a6cbe4a171479a9b871723789b3101 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:46.258669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:46.311993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:46.316722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:46.368092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4722b4-7ec3-42e7-a90e-ead0f9d61643 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:46.377270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b050c3-86f6-4dae-b2d6-55df354d58ac None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:46.405782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:46.406384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.406653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.406921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.407730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, user.id=07a6cbe4a171479a9b871723789b3101, scope.system=all) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:46.408035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:46.409733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:46.414944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 140/280] 10.222.0.33 () {56 vars in 1308 bytes} [Thu Oct 6 04:18:46 2022] GET /identity/v3/role_assignments?user.id=07a6cbe4a171479a9b871723789b3101&scope.system=all&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 226 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:46.421397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b646cb4-cb5b-4475-b347-3515b8e2dbe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:46.422360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b646cb4-cb5b-4475-b347-3515b8e2dbe6 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.422856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b646cb4-cb5b-4475-b347-3515b8e2dbe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.423448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b646cb4-cb5b-4475-b347-3515b8e2dbe6 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.424350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b646cb4-cb5b-4475-b347-3515b8e2dbe6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:46.426060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b646cb4-cb5b-4475-b347-3515b8e2dbe6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:46.455352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 141/281] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:18:46 2022] POST /identity/v3/users => generated 293 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:46.507098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:46.643205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd3dd6f-f3a6-4944-9327-6b05e3e87cc8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:46.659109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:46.681733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c386fa-2280-4c6e-82b6-4fd663266f3c None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.682245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c386fa-2280-4c6e-82b6-4fd663266f3c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.682245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c386fa-2280-4c6e-82b6-4fd663266f3c None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.683638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 142/282] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:46.685693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:46.686546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.686993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.687403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.689415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f8b7c-a9f6-41f9-81ec-53e8a9955cf2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.690018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f8b7c-a9f6-41f9-81ec-53e8a9955cf2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.690403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f8b7c-a9f6-41f9-81ec-53e8a9955cf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.698721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:46.704605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:46.705520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:46.706322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 141/283] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:46.711910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:46.713634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-059f8b7c-a9f6-41f9-81ec-53e8a9955cf2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:46.714818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-059f8b7c-a9f6-41f9-81ec-53e8a9955cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:46.807967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 143/284] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:46.844454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b117b72-990d-4573-8395-44b6d1006400 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:46.886220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:46.886824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:46.887160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:46.887436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:46.888697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:46.889026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:46.890486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:46.899734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 142/285] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:46 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:46.905231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059f8b7c-a9f6-41f9-81ec-53e8a9955cf2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:46.953078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:47.017074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059f8b7c-a9f6-41f9-81ec-53e8a9955cf2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:47.058698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:47.060226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:47.060710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:47.061177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] PATH_INFO: `/v3/users/191f3e755aa64b249e73ee2f700f6cee` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:47.067662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] RBAC: Authorizing `identity:get_user(user_id=191f3e755aa64b249e73ee2f700f6cee)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:47.069350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:47.071511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 144/286] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:46 2022] GET /identity/v3/users/191f3e755aa64b249e73ee2f700f6cee => generated 297 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:47.077082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:47.121400 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:47.125176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:47.243834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298eca57-6493-4be3-b64b-9d094e8119f4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:47.279690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:47.280342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:47.280624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:47.280898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:47.282132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:47.283695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:47.285327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 143/287] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:47 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:47.292734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:47.323626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5b9258-ff0a-4be4-92cd-214c3cb6f150 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:47.335470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:47.389121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:47.389764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:47.390039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:47.390312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:47.415222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:47.422239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:47.423143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:47.423923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:47.429041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 145/288] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:18:47.435403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:47.485132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbe6b73-bd85-44a3-a221-00d502d1c5ec None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:47.487674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:47.554491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:47.555845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:47.556143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:47.556419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47/users/191f3e755aa64b249e73ee2f700f6cee/roles/5cef7e0c81c54474b23931f34295ad6e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:47.560496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] RBAC: Authorizing `identity:create_grant(project_id=5698e9901f634a3695852e6b8b460d47, user_id=191f3e755aa64b249e73ee2f700f6cee, role_id=5cef7e0c81c54474b23931f34295ad6e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:47.561949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:47.582901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 144/289] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:18:47 2022] PUT /identity/v3/projects/5698e9901f634a3695852e6b8b460d47/users/191f3e755aa64b249e73ee2f700f6cee/roles/5cef7e0c81c54474b23931f34295ad6e => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:18:47.612570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a1402-4147-417e-837f-bfb2c0534510 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:47.660609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:47.661545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:47.662030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:47.662468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:47.663479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:47.663948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:47.665488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:47.680272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 146/290] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:47.686717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:47.837732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda75c4f-2455-4652-8e15-3f25f8a2a30a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:47.875868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:47.876501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:47.876775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:47.877049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] PATH_INFO: `/v3/users/07320404c7be4223a89211cb9888a17e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:47.878168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] RBAC: Authorizing `identity:get_user(user_id=07320404c7be4223a89211cb9888a17e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:47.879625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:47.881270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 145/291] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:47 2022] GET /identity/v3/users/07320404c7be4223a89211cb9888a17e => generated 297 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:47.887149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:47.920238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:48.006305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4505fa1d-5044-4495-a816-f4a946a681a5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:48.052063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:48.052673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:48.053009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:48.053285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:48.142941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:48.145185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:48.184597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:48.294058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 147/292] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:48.300159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:48.360970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:48.459542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049e03e9-a6fe-4c97-8be7-dcfe98bf0e1b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:48.505082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006ac92c-626a-427d-a33f-d3710475f88c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:48.505805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006ac92c-626a-427d-a33f-d3710475f88c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:48.506079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006ac92c-626a-427d-a33f-d3710475f88c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:48.506351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006ac92c-626a-427d-a33f-d3710475f88c None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:48.507607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006ac92c-626a-427d-a33f-d3710475f88c None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:48.509117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006ac92c-626a-427d-a33f-d3710475f88c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:48.510985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 146/293] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:48 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:48.517905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:48.642380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7dba5a-96c1-424f-85f9-77e0ba68a4a9 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:48.684560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:48.691483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33df7be-4d89-400c-86d1-a2363bbfdffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:48.692141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33df7be-4d89-400c-86d1-a2363bbfdffb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:48.692416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33df7be-4d89-400c-86d1-a2363bbfdffb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:48.692682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33df7be-4d89-400c-86d1-a2363bbfdffb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:48.693414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33df7be-4d89-400c-86d1-a2363bbfdffb None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=07320404c7be4223a89211cb9888a17e) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:48.693818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33df7be-4d89-400c-86d1-a2363bbfdffb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:48.695363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33df7be-4d89-400c-86d1-a2363bbfdffb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:48.707875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 148/294] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:48 2022] GET /identity/v3/role_assignments?user.id=07320404c7be4223a89211cb9888a17e&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:49.260609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a414733-282e-49e8-8cd8-cff1fe77dc2b None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.261446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a414733-282e-49e8-8cd8-cff1fe77dc2b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.261858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a414733-282e-49e8-8cd8-cff1fe77dc2b None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.263565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 147/295] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:49.269311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b7e6a4-e31a-419a-b44e-191773bba84f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.270173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b7e6a4-e31a-419a-b44e-191773bba84f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.270618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b7e6a4-e31a-419a-b44e-191773bba84f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.291914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-08b7e6a4-e31a-419a-b44e-191773bba84f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:49.293534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-08b7e6a4-e31a-419a-b44e-191773bba84f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:49.375147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 149/296] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:49.424093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6cc3c3-7737-4ffb-ac12-0f373d123cc4 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.424399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6cc3c3-7737-4ffb-ac12-0f373d123cc4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.424627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6cc3c3-7737-4ffb-ac12-0f373d123cc4 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.425964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 148/297] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:49.431171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022eb12f-16b7-4198-b6b5-de58c906ceed None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.431652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022eb12f-16b7-4198-b6b5-de58c906ceed None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.432087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022eb12f-16b7-4198-b6b5-de58c906ceed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.452543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-022eb12f-16b7-4198-b6b5-de58c906ceed None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:49.453988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-022eb12f-16b7-4198-b6b5-de58c906ceed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:49.489176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd5f0a7-2134-4c45-acff-6e77088813bc None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.489484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd5f0a7-2134-4c45-acff-6e77088813bc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.489645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd5f0a7-2134-4c45-acff-6e77088813bc None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.490990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 149/298] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:49.496930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a34d1e-1f3e-4fd1-be30-88910c775a4c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.497771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a34d1e-1f3e-4fd1-be30-88910c775a4c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.498039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a34d1e-1f3e-4fd1-be30-88910c775a4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.517278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-56a34d1e-1f3e-4fd1-be30-88910c775a4c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:49.518592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-56a34d1e-1f3e-4fd1-be30-88910c775a4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:49.529453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 150/299] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:49.599093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 150/300] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:49.663189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022eb12f-16b7-4198-b6b5-de58c906ceed None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:49.722164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:49.773758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022eb12f-16b7-4198-b6b5-de58c906ceed None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:49.809000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:49.809710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.809949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.810215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.817983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a34d1e-1f3e-4fd1-be30-88910c775a4c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:49.820909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:49.826783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:49.827682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:49.828470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:49.830425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 151/301] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:49.836171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:49.855863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:49.936360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a34d1e-1f3e-4fd1-be30-88910c775a4c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:49.949432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978d7191-e8fb-4597-a458-ef72314a1832 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:49.965238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:49.972196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:49.972825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.973097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.973368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.974454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:49.976079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:49.979006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 151/302] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:49.979006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:49.991176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:49.991778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:49.992055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:49.992325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:49.992986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:49.993286 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:49.994660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:50.003362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 152/303] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:50.004382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:50.084645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:50.091922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38f38e-6363-45af-a5b6-04fbc4826418 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.112649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9477f225-e43d-41fc-b3c8-45cf5f351a85 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.129081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.129717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.130155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.130568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.143160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:50.148783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:50.149958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:50.151082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:50.151846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.151846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.151846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.151846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] PATH_INFO: `/v3/groups` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.152150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:50.152345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:50.154664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 152/304] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:50.155825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:50.177061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 153/305] 10.222.0.33 () {60 vars in 1134 bytes} [Thu Oct 6 04:18:50 2022] POST /identity/v3/groups => generated 229 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:50.177882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:50.236331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:50.305305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb08f4a8-22b8-4f85-a77d-61408c7d0015 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.320641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56eda5a9-e076-40ad-b90d-4ab529b1557a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.346119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37e4889-e59f-4270-a298-d7df1a30b048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.347026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37e4889-e59f-4270-a298-d7df1a30b048 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.347454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37e4889-e59f-4270-a298-d7df1a30b048 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.347723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37e4889-e59f-4270-a298-d7df1a30b048 None admin] PATH_INFO: `/v3/users/07a6cbe4a171479a9b871723789b3101` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.348979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37e4889-e59f-4270-a298-d7df1a30b048 None admin] RBAC: Authorizing `identity:get_user(user_id=07a6cbe4a171479a9b871723789b3101)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:50.350478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37e4889-e59f-4270-a298-d7df1a30b048 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:50.357342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 153/306] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:50 2022] GET /identity/v3/users/07a6cbe4a171479a9b871723789b3101 => generated 293 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:50.357342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4f9be-a5dd-41f1-bd0a-e225de3f0bae None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.357342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4f9be-a5dd-41f1-bd0a-e225de3f0bae None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.357342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4f9be-a5dd-41f1-bd0a-e225de3f0bae None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.357342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 154/307] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:50 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:50.360481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4f9be-a5dd-41f1-bd0a-e225de3f0bae None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:50.366843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2097461-a831-45c5-84aa-18669b8513b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.367566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2097461-a831-45c5-84aa-18669b8513b4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.367853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2097461-a831-45c5-84aa-18669b8513b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.368119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2097461-a831-45c5-84aa-18669b8513b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.368791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2097461-a831-45c5-84aa-18669b8513b4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:50.369093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2097461-a831-45c5-84aa-18669b8513b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:50.370465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2097461-a831-45c5-84aa-18669b8513b4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:50.380381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 154/308] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:50.381625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc4857-2f01-46ee-bc7c-ae041970e494 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.382776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc4857-2f01-46ee-bc7c-ae041970e494 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.383078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc4857-2f01-46ee-bc7c-ae041970e494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.389994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:50.403316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f6bc4857-2f01-46ee-bc7c-ae041970e494 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:50.405112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f6bc4857-2f01-46ee-bc7c-ae041970e494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:50.473764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 155/309] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:50.474617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc4857-2f01-46ee-bc7c-ae041970e494 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:50.485841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4f9be-a5dd-41f1-bd0a-e225de3f0bae None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.504752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:50.530077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.531120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.531593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.532056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] PATH_INFO: `/v3/system/users/07a6cbe4a171479a9b871723789b3101/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.533727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=07a6cbe4a171479a9b871723789b3101, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:50.535604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:50.546693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 155/310] 10.222.0.33 () {58 vars in 1334 bytes} [Thu Oct 6 04:18:50 2022] PUT /identity/v3/system/users/07a6cbe4a171479a9b871723789b3101/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:18:50.590179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc4857-2f01-46ee-bc7c-ae041970e494 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.638669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.639278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.639553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.639821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] PATH_INFO: `/v3/users/24416cddd8174632a3ef736d69595b4c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.655969 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] RBAC: Authorizing `identity:get_user(user_id=24416cddd8174632a3ef736d69595b4c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:50.657584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:50.659440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 156/311] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:50 2022] GET /identity/v3/users/24416cddd8174632a3ef736d69595b4c => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:50.665245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:50.756835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:50.770607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:50.784965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b457daf8-c8fc-491e-bac4-d1e2b05baa32 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.821332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.821956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.822230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.822543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.823293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] RBAC: Adding query filter params (role.id=5cef7e0c81c54474b23931f34295ad6e, user.id=24416cddd8174632a3ef736d69595b4c, scope.system=all) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:50.823594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:50.825005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:50.830210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 156/312] 10.222.0.33 () {56 vars in 1308 bytes} [Thu Oct 6 04:18:50 2022] GET /identity/v3/role_assignments?user.id=24416cddd8174632a3ef736d69595b4c&scope.system=all&role.id=5cef7e0c81c54474b23931f34295ad6e => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:50.881031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:50.901485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a9227-c6bc-47f7-8cb8-7504e4f95a7d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:50.941260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:50.941958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:50.942231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:50.942500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:50.954889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:50.959905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:50.960852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:50.961645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 157/313] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:18:50.967890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:51.081118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f86da1-87a6-4869-a9b9-e6fca22e9d76 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:51.117649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:51.118249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:51.118529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:51.118852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:51.119553 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:51.119849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:51.121197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:51.130132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 157/314] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:51.135758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:51.249241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2563b63-3ad0-4792-9f33-ae193cba54d4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:51.285171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:51.286056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:51.286471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:51.286879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] PATH_INFO: `/v3/users/191f3e755aa64b249e73ee2f700f6cee` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:51.288162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] RBAC: Authorizing `identity:get_user(user_id=191f3e755aa64b249e73ee2f700f6cee)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:51.289746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:51.291559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 158/315] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:51 2022] GET /identity/v3/users/191f3e755aa64b249e73ee2f700f6cee => generated 297 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:51.302995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:51.314812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:51.406443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b336ad4-75a1-48b6-a50b-eda2e8e18471 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:51.440515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:51.441115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:51.441393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:51.441657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:51.513005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:51.514667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:51.530317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:51.619981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 158/316] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:51.628061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:51.644233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:51.738610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2286e71-26ef-4b2e-8db7-a364e002a0d5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:51.775755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1659125-2a7b-4ffe-a78b-425877029349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:51.776693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1659125-2a7b-4ffe-a78b-425877029349 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:51.777170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1659125-2a7b-4ffe-a78b-425877029349 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:51.777637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1659125-2a7b-4ffe-a78b-425877029349 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:51.779975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1659125-2a7b-4ffe-a78b-425877029349 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:51.781633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1659125-2a7b-4ffe-a78b-425877029349 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:51.783510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 159/317] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:51 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:51.791574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:51.899010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de33b2d-faaa-4df4-989f-548dec70df85 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:51.916459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:51.938786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfa9afe-ef37-4a98-a949-eb2b61a07a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:51.939413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa9afe-ef37-4a98-a949-eb2b61a07a52 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:51.939685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa9afe-ef37-4a98-a949-eb2b61a07a52 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:51.939953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa9afe-ef37-4a98-a949-eb2b61a07a52 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:51.940664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfa9afe-ef37-4a98-a949-eb2b61a07a52 None admin] RBAC: Adding query filter params (role.id=5cef7e0c81c54474b23931f34295ad6e, scope.project.id=5698e9901f634a3695852e6b8b460d47, user.id=191f3e755aa64b249e73ee2f700f6cee) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:51.940963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfa9afe-ef37-4a98-a949-eb2b61a07a52 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:51.942563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfa9afe-ef37-4a98-a949-eb2b61a07a52 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:51.948779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 159/318] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:18:51 2022] GET /identity/v3/role_assignments?user.id=191f3e755aa64b249e73ee2f700f6cee&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=5cef7e0c81c54474b23931f34295ad6e => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:52.507851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fbb835-0056-4fcf-843f-6e6887479276 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:52.508185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fbb835-0056-4fcf-843f-6e6887479276 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:52.508360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fbb835-0056-4fcf-843f-6e6887479276 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:52.509695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 160/319] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:52.514980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff11088-8379-4958-9182-2eb70154cba8 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:52.515417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff11088-8379-4958-9182-2eb70154cba8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:52.515577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff11088-8379-4958-9182-2eb70154cba8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:52.534248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bff11088-8379-4958-9182-2eb70154cba8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:52.535469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bff11088-8379-4958-9182-2eb70154cba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:52.607001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 160/320] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:52.809725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa662fd-31fc-4a1b-9198-f284755fb3a8 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:52.810213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa662fd-31fc-4a1b-9198-f284755fb3a8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:52.810213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa662fd-31fc-4a1b-9198-f284755fb3a8 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:52.811602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 161/321] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:52.816827 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b40836-985c-473a-b00a-b86aec373211 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:52.817123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b40836-985c-473a-b00a-b86aec373211 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:52.817402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b40836-985c-473a-b00a-b86aec373211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:52.835611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-21b40836-985c-473a-b00a-b86aec373211 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:52.836792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-21b40836-985c-473a-b00a-b86aec373211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:52.873903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa662fd-31fc-4a1b-9198-f284755fb3a8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:52.908719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 161/322] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:52.957388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:52.983603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa662fd-31fc-4a1b-9198-f284755fb3a8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.020960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731d0ae9-1658-4c4b-bcdc-6aeb9867f795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.021872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d0ae9-1658-4c4b-bcdc-6aeb9867f795 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.022146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d0ae9-1658-4c4b-bcdc-6aeb9867f795 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.022414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d0ae9-1658-4c4b-bcdc-6aeb9867f795 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.023556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731d0ae9-1658-4c4b-bcdc-6aeb9867f795 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:53.025198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731d0ae9-1658-4c4b-bcdc-6aeb9867f795 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:53.027187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 162/323] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:18:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:53.032695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b40836-985c-473a-b00a-b86aec373211 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:53.078089 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:53.138604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b40836-985c-473a-b00a-b86aec373211 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.157213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936a284-fdd7-459b-911a-6d6e4563e348 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.157516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936a284-fdd7-459b-911a-6d6e4563e348 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.157767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936a284-fdd7-459b-911a-6d6e4563e348 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.159889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 163/324] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:53.164676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b1616-fc90-4163-abb0-8deefeb4a9df None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.164956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b1616-fc90-4163-abb0-8deefeb4a9df None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.165226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b1616-fc90-4163-abb0-8deefeb4a9df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.173156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.173751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.174023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.174289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] PATH_INFO: `/v3/groups` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.174915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:53.176427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:53.183294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c44b1616-fc90-4163-abb0-8deefeb4a9df None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:53.185123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c44b1616-fc90-4163-abb0-8deefeb4a9df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:53.193860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 162/325] 10.222.0.33 () {60 vars in 1134 bytes} [Thu Oct 6 04:18:53 2022] POST /identity/v3/groups => generated 226 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:18:53.194856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:53.275016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 164/326] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:53.311873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:53.318879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0290c275-ea43-489e-a805-e3c2f8194288 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.365784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.366631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.367080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.367495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.383361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:53.389782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:53.390673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:53.391568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 163/327] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:53.396827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44b1616-fc90-4163-abb0-8deefeb4a9df None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:53.479851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:53.505046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44b1616-fc90-4163-abb0-8deefeb4a9df None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.544762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.545751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.546247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.546586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:53.547278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.548136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:53.548627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:53.550198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:53.559957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 165/328] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:53.567773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:53.689293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7032bbca-b008-46bf-af49-4ae12cb81bc9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.695429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1832d173-3d55-436b-a79a-526700b3013d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.732626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.732626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.732887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.732982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.737206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.737844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.738116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.738382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] PATH_INFO: `/v3/users/07a6cbe4a171479a9b871723789b3101` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.739674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] RBAC: Authorizing `identity:get_user(user_id=07a6cbe4a171479a9b871723789b3101)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:53.741215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:53.744991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:53.749372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 166/329] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:53 2022] GET /identity/v3/users/07a6cbe4a171479a9b871723789b3101 => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:53.750253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:53.751147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:53.752037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:53.752867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 164/330] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:53 2022] GET /identity/v3/roles/reader => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:53.758581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:53.769778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:53.779319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:53.888976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c4ed3b-f2fb-420a-a6a9-41c33bfe591f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.895599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a5a264-9bd5-469f-b14a-1906b8367280 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:53.932184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.932826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.933102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.933372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.934305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] RBAC: Adding query filter params (role.id=68ada91d9625461ea199ad5a616e8f01, user.id=07a6cbe4a171479a9b871723789b3101, scope.system=all) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:53.937644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:53.939517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:53.941666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:53.943056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:53.943817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:53.944290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:53.945414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:53.946284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:53.947908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:53.958526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 165/332] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:53 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:18:53.960044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 167/332] 10.222.0.33 () {56 vars in 1308 bytes} [Thu Oct 6 04:18:53 2022] GET /identity/v3/role_assignments?user.id=07a6cbe4a171479a9b871723789b3101&scope.system=all&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 510 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:53.963987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:54.070403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:54.093203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b4fc3e-a6b0-4d6e-a6db-0d403b67ef53 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:54.132580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa09464b-4b9c-4124-9e83-4034218d547f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:54.133583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa09464b-4b9c-4124-9e83-4034218d547f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:54.134064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa09464b-4b9c-4124-9e83-4034218d547f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:54.134540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa09464b-4b9c-4124-9e83-4034218d547f None admin] PATH_INFO: `/v3/users/24416cddd8174632a3ef736d69595b4c` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:54.136137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa09464b-4b9c-4124-9e83-4034218d547f None admin] RBAC: Authorizing `identity:get_user(user_id=24416cddd8174632a3ef736d69595b4c)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:54.137957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa09464b-4b9c-4124-9e83-4034218d547f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:54.140404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 166/333] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:53 2022] GET /identity/v3/users/24416cddd8174632a3ef736d69595b4c => generated 293 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:54.145745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:54.220220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:54.257442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6f930d-0d98-428c-9449-63bf07c052d0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:54.294282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093f6fe9-408d-4f69-9562-d9d920736e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:54.294995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f6fe9-408d-4f69-9562-d9d920736e39 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:54.295372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f6fe9-408d-4f69-9562-d9d920736e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:54.295652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f6fe9-408d-4f69-9562-d9d920736e39 None admin] PATH_INFO: `/v3/system/users/24416cddd8174632a3ef736d69595b4c/roles/5cef7e0c81c54474b23931f34295ad6e` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:54.297096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093f6fe9-408d-4f69-9562-d9d920736e39 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=24416cddd8174632a3ef736d69595b4c, role_id=5cef7e0c81c54474b23931f34295ad6e)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:54.298482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093f6fe9-408d-4f69-9562-d9d920736e39 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:54.309870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 168/334] 10.222.0.33 () {58 vars in 1334 bytes} [Thu Oct 6 04:18:54 2022] PUT /identity/v3/system/users/24416cddd8174632a3ef736d69595b4c/roles/5cef7e0c81c54474b23931f34295ad6e => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:18:56.130275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee026f90-13e5-4b37-bb70-97b41a510c6b None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:56.131303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee026f90-13e5-4b37-bb70-97b41a510c6b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:56.131785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee026f90-13e5-4b37-bb70-97b41a510c6b None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:56.133607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 167/335] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:56.140191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476e287-f8f7-47be-aa51-921b08a1a045 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:56.141022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476e287-f8f7-47be-aa51-921b08a1a045 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:56.141530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476e287-f8f7-47be-aa51-921b08a1a045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:56.163166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3476e287-f8f7-47be-aa51-921b08a1a045 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:56.164683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3476e287-f8f7-47be-aa51-921b08a1a045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:56.261613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 169/336] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:56.733900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fca124-9cce-44a3-832c-10d5f0283aaa None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:56.734556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fca124-9cce-44a3-832c-10d5f0283aaa None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:56.735059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fca124-9cce-44a3-832c-10d5f0283aaa None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:56.736815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 168/337] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:56.744134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad6d552-87ae-4fd6-8433-936fb907797d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:56.744712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad6d552-87ae-4fd6-8433-936fb907797d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:56.745192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad6d552-87ae-4fd6-8433-936fb907797d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:56.769670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3ad6d552-87ae-4fd6-8433-936fb907797d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:56.770890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3ad6d552-87ae-4fd6-8433-936fb907797d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:56.796764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fca124-9cce-44a3-832c-10d5f0283aaa None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:56.852575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:56.867474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 170/338] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:56.984266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fca124-9cce-44a3-832c-10d5f0283aaa None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:57.026797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:57.027846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.028873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.029361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.045836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:57.051651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:57.052615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:57.053404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:57.056194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 169/339] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:57.062058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad6d552-87ae-4fd6-8433-936fb907797d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:57.184680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad6d552-87ae-4fd6-8433-936fb907797d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:57.230472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47290af7-5139-481a-885f-ca38e9ba8a5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:57.231676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47290af7-5139-481a-885f-ca38e9ba8a5c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.231968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47290af7-5139-481a-885f-ca38e9ba8a5c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.232232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47290af7-5139-481a-885f-ca38e9ba8a5c None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.232907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47290af7-5139-481a-885f-ca38e9ba8a5c None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:57.233210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47290af7-5139-481a-885f-ca38e9ba8a5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:57.234664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47290af7-5139-481a-885f-ca38e9ba8a5c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:57.243998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 171/340] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:57.249926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:57.254707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a1775-9e03-4ea6-b38f-469df864b1b6 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.255079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a1775-9e03-4ea6-b38f-469df864b1b6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.255361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8a1775-9e03-4ea6-b38f-469df864b1b6 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.257024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 172/341] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:57.268173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab22ce3-ab46-43f9-add7-53b213659506 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.268509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab22ce3-ab46-43f9-add7-53b213659506 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.268720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab22ce3-ab46-43f9-add7-53b213659506 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.270427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 173/342] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:18:57.271950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac63493-b98d-48e1-937e-04fc3c8652c2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.272237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac63493-b98d-48e1-937e-04fc3c8652c2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.272510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac63493-b98d-48e1-937e-04fc3c8652c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.273717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:57.316471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6ac63493-b98d-48e1-937e-04fc3c8652c2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:57.317826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6ac63493-b98d-48e1-937e-04fc3c8652c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:57.417106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:57.435030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890ed301-9a75-485b-b237-4822d9e74c8f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:57.439991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 174/343] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:18:57.441135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb51060-6d52-436d-905e-66bb1b77c3dd None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.441422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb51060-6d52-436d-905e-66bb1b77c3dd None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.441693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb51060-6d52-436d-905e-66bb1b77c3dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.474071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f995543-05f8-41e4-be5b-efb89bc071dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:57.476127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f995543-05f8-41e4-be5b-efb89bc071dd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.476661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f995543-05f8-41e4-be5b-efb89bc071dd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.477114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f995543-05f8-41e4-be5b-efb89bc071dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.477373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5cb51060-6d52-436d-905e-66bb1b77c3dd None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:57.478622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5cb51060-6d52-436d-905e-66bb1b77c3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:57.561337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 175/344] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:57.562260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb51060-6d52-436d-905e-66bb1b77c3dd None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:57.563499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f995543-05f8-41e4-be5b-efb89bc071dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:57.565283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f995543-05f8-41e4-be5b-efb89bc071dd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:57.579186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:57.721232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb51060-6d52-436d-905e-66bb1b77c3dd None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:57.730723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 170/345] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:57.733091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd786b-ef4d-468a-b1fa-ae1fda587122 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.733674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd786b-ef4d-468a-b1fa-ae1fda587122 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.734146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cd786b-ef4d-468a-b1fa-ae1fda587122 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.735725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 171/346] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:57.740941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cd786b-ef4d-468a-b1fa-ae1fda587122 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:57.741472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:57.763431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a7399-7297-4493-8f66-0c0a0aa0298f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:57.764340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a7399-7297-4493-8f66-0c0a0aa0298f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.764797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a7399-7297-4493-8f66-0c0a0aa0298f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.765223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a7399-7297-4493-8f66-0c0a0aa0298f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.777357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61a7399-7297-4493-8f66-0c0a0aa0298f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a61a7399-7297-4493-8f66-0c0a0aa0298f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:57.783729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:57.784657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 06 04:18:57.785496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:57.787831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 176/347] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:57.789705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ee312-2502-4d49-a823-39971c39c04b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.790198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ee312-2502-4d49-a823-39971c39c04b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.790677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096ee312-2502-4d49-a823-39971c39c04b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.815211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-096ee312-2502-4d49-a823-39971c39c04b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:57.816722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-096ee312-2502-4d49-a823-39971c39c04b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:57.883389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cd786b-ef4d-468a-b1fa-ae1fda587122 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:57.891898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 177/348] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:57.893033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ee312-2502-4d49-a823-39971c39c04b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:57.903552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:57.943317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:57.943940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:57.944373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:57.944787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:57.951780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:57.953488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:57.955477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 172/349] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:57 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:57.956393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.007938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096ee312-2502-4d49-a823-39971c39c04b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.011370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:58.048875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.050411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.050854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.051361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.065726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:58.072051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:58.073177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:58.074203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:18:58.077756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 178/350] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:58.078719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.100776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf9629-c106-4361-9e4e-1c51ebeff4bd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.142926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.143601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.143880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.144174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.144819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:58.145121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:58.146492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:58.157433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 173/351] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:58.158476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.204820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:58.211187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125dc502-089e-4d0b-9b17-fbf2fd40afe0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.250168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.251503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.252236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.252675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.260699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:58.263031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:58.269896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 179/352] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:58.270897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.279747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59ec41-6fd8-4777-9e30-ddb4d2013eb7 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.310364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:58.329029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.330022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.330359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.330822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.341822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:58.347023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:58.348085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:18:58.349074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:58.351020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 174/353] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:58.351982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.409832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cfefeb-636e-4442-9203-1e5529b31a8e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.463583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.464248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.464706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.465149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.466659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:58.466996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:58.468419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:58.481238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 180/354] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:58.483356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.484526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:58.489979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48156b90-cbd5-4c26-97fc-ba0ab144ef54 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.531059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.531960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.532234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.532505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] PATH_INFO: `/v3/users/24416cddd8174632a3ef736d69595b4c` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.533724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] RBAC: Authorizing `identity:get_user(user_id=24416cddd8174632a3ef736d69595b4c)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:58.535289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:58.537067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 175/355] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/users/24416cddd8174632a3ef736d69595b4c => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:58.537661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.604101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:58.615641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70e8797-9b97-4bbf-82ff-9839952a6fc3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.654261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.655329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.655994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.656279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.657545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:58.658188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:58.661061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:58.666819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3173a5ef-4723-4d89-9471-42cf199e3959 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.669812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 181/356] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 260 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:58.670883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.711021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.711658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.711936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.712403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.731962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:18:58.740228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:18:58.741390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:18:58.742301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:18:58.743532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 176/357] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:18:58.743609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.788973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:58.811224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b01b56-6629-4cfe-bb26-b90581ca51b2 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.865822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.866416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.866749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.867103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:58.867771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:58.868078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:58.869474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:58.879352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 182/358] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:58.880186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:58.901431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f44593c-6ffd-4556-bdb1-3ef6ff764f65 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:58.902543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:58.960891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:58.961491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:58.961764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:58.962028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.009012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b55a99-76ba-4e1f-923a-1abd0242c2e6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:59.054624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.055708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af5fd5e-93df-46fe-bf11-2b9086cc358f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:59.056111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.056749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af5fd5e-93df-46fe-bf11-2b9086cc358f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.057247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af5fd5e-93df-46fe-bf11-2b9086cc358f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.057723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af5fd5e-93df-46fe-bf11-2b9086cc358f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.058651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af5fd5e-93df-46fe-bf11-2b9086cc358f None admin] RBAC: Adding query filter params (role.id=5cef7e0c81c54474b23931f34295ad6e, user.id=24416cddd8174632a3ef736d69595b4c, scope.system=all) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:59.059852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af5fd5e-93df-46fe-bf11-2b9086cc358f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.061548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af5fd5e-93df-46fe-bf11-2b9086cc358f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.074646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 183/359] 10.222.0.33 () {56 vars in 1308 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/role_assignments?user.id=24416cddd8174632a3ef736d69595b4c&scope.system=all&role.id=5cef7e0c81c54474b23931f34295ad6e => generated 510 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:59.076330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5518e172-f2fa-4495-a5a2-9db792800a37 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.076612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5518e172-f2fa-4495-a5a2-9db792800a37 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.076939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5518e172-f2fa-4495-a5a2-9db792800a37 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.078217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 184/360] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:18:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:18:59.078951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5518e172-f2fa-4495-a5a2-9db792800a37 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:59.099000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:59.195060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:59.211334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 177/361] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:59.214501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:59.243590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5518e172-f2fa-4495-a5a2-9db792800a37 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:59.287526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16367511-ffaf-4e1b-b598-ffa7f4474f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:59.288526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16367511-ffaf-4e1b-b598-ffa7f4474f6c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.288991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16367511-ffaf-4e1b-b598-ffa7f4474f6c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.289433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16367511-ffaf-4e1b-b598-ffa7f4474f6c None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.290281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16367511-ffaf-4e1b-b598-ffa7f4474f6c None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:18:59.290780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16367511-ffaf-4e1b-b598-ffa7f4474f6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.292460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16367511-ffaf-4e1b-b598-ffa7f4474f6c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.305815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 185/362] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:59.307457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdeef6d-f0c9-4544-9f4b-44e784aafbd2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.307910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdeef6d-f0c9-4544-9f4b-44e784aafbd2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.308374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdeef6d-f0c9-4544-9f4b-44e784aafbd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.329141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-0fdeef6d-f0c9-4544-9f4b-44e784aafbd2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:18:59.330650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0fdeef6d-f0c9-4544-9f4b-44e784aafbd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:18:59.360694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b0a948-586e-444c-9a57-c366e8a1404a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:59.400887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:59.416339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:59.416962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.417240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.417513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.423279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 186/363] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:18:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:18:59.424160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdeef6d-f0c9-4544-9f4b-44e784aafbd2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:59.494865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.496570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.502133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:59.547179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdeef6d-f0c9-4544-9f4b-44e784aafbd2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:59.593811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:59.594460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.594735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.595023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.596243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.597736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.600870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 187/364] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:59.601721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:59.611154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 178/365] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:59.612530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:59.645328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:59.730139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89cad5d-ec6f-46d0-93ff-6b3347021e94 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:59.745315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2371313e-d964-4d29-acf3-1e90a81b8900 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:59.775522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:59.776330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.776611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.776885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.780388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:59.792966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:59.794222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.794684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.794952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.796078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.797515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.799107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 179/366] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:59.799938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:59.860810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.862862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.917236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:18:59.923313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b888d171-7348-4c4b-ac57-74dbfebcff77 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:18:59.965178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:18:59.966368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:18:59.966785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:18:59.967022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:18:59.968179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:18:59.969615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:18:59.972008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 180/367] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:18:59.973133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:18:59.978487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 188/368] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:18:59.980061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.018061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:00.109726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c42bf-c39d-45e6-bade-72ce1ec2c0ec None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.115384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1689fdeb-d34c-499a-b833-68f7b89fa584 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.149779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.150778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.151275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.151736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.154917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.155806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.156271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.156728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.157693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=5698e9901f634a3695852e6b8b460d47) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:00.158246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:00.159899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:00.166647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:00.167420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 189/369] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:00.169177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:00.173145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:00.174102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:00.174938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 181/370] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:18:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:00.175717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.212496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:00.309758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:00.318305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4754634e-d554-44a5-aaff-24ff1ccd87da None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.331518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d02dfa-b2cc-4ac3-872b-1a74d110258e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.366853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.367919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.368493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.368965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.370982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:00.373677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:00.376537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 190/371] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:00 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:00.377768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.385076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.385789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.386136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.386479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.388380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:00.389887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:00.393148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 182/372] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:00 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:00.393908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.510173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f3ac80-2a32-4760-a327-e35473e2db78 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.510401 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:00.524532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db813a1a-4ff7-4040-8e7d-88ac50be7e5c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.548001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.548609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.548885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.549153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.549921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:00.550222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:00.551613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:00.562733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 191/373] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:00.563836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.566169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.566880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.567246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.567519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.568304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:00.568616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:00.570043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:00.576984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 183/374] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:00 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 260 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:00.586579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.598818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:00.730402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fd6f64-2d66-4045-ae53-f1f091856174 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.750385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86fafea-38da-462e-bdf7-b3d58fedc09c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:00.781459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125617bf-1632-4911-8c95-a72d1bdd49ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.782126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125617bf-1632-4911-8c95-a72d1bdd49ee None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.782411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125617bf-1632-4911-8c95-a72d1bdd49ee None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.782741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125617bf-1632-4911-8c95-a72d1bdd49ee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.835889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:00.836547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:00.836825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:00.837096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:00.838922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:00.840475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:00.846688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 192/375] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:00 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:00.852214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:00.855343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:00.934412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125617bf-1632-4911-8c95-a72d1bdd49ee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:00.936413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125617bf-1632-4911-8c95-a72d1bdd49ee None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:01.003295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:01.010266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92a9cc3-2917-4d80-9ea6-1c34563ab3fb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:01.060441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:01.061255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:01.061639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:01.062018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:01.062865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=5698e9901f634a3695852e6b8b460d47) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:01.063309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:01.064883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:01.072433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 193/376] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:00 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 260 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:01.089232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 184/377] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 504 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 06 04:19:01.095637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:01.214013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0585d25-fe0a-4e9d-9cad-cc9efcf8759d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:01.254405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e17ff-a6e1-42e0-850f-4528dc4cd719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:01.255094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e17ff-a6e1-42e0-850f-4528dc4cd719 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:01.255377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e17ff-a6e1-42e0-850f-4528dc4cd719 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:01.255649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e17ff-a6e1-42e0-850f-4528dc4cd719 None admin] PATH_INFO: `/v3/projects/7760d317949542fd9d86e529eca476cc` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:01.259665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592e17ff-a6e1-42e0-850f-4528dc4cd719 None admin] RBAC: Authorizing `identity:get_project(project_id=7760d317949542fd9d86e529eca476cc)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:01.260738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254a2f0-adcb-4c05-9bce-76c16e066b12 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:01.261034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254a2f0-adcb-4c05-9bce-76c16e066b12 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:01.261310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254a2f0-adcb-4c05-9bce-76c16e066b12 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:01.261414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592e17ff-a6e1-42e0-850f-4528dc4cd719 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:01.263011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 185/378] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:19:01.263404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 194/379] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:01 2022] GET /identity/v3/projects/7760d317949542fd9d86e529eca476cc => generated 344 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:01.268954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a254a2f0-adcb-4c05-9bce-76c16e066b12 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:01.273934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88650952-2801-4c25-a1b8-c006e54a25e2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:01.274228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88650952-2801-4c25-a1b8-c006e54a25e2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:01.274506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88650952-2801-4c25-a1b8-c006e54a25e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:01.307358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-88650952-2801-4c25-a1b8-c006e54a25e2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:01.309036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-88650952-2801-4c25-a1b8-c006e54a25e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:01.350928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:01.353121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:01.411468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a254a2f0-adcb-4c05-9bce-76c16e066b12 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:01.429104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 195/380] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:01.453653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:01.454280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:01.454551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:01.454817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] PATH_INFO: `/v3/groups/f28434a0fc2e44c78ff390df07003ed5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:01.461151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] RBAC: Authorizing `identity:get_group(group_id=f28434a0fc2e44c78ff390df07003ed5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:01.462633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:01.464566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 186/381] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:01 2022] GET /identity/v3/groups/f28434a0fc2e44c78ff390df07003ed5 => generated 229 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:01.472261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88650952-2801-4c25-a1b8-c006e54a25e2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:01.616757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88650952-2801-4c25-a1b8-c006e54a25e2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:01.667661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:01.668273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:01.669074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:01.669445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:01.670064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] RBAC: Adding query filter params (group.id=f28434a0fc2e44c78ff390df07003ed5, role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=7760d317949542fd9d86e529eca476cc) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:01.670365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:01.671809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:01.677857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 196/382] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:01 2022] GET /identity/v3/role_assignments?group.id=f28434a0fc2e44c78ff390df07003ed5&scope.project.id=7760d317949542fd9d86e529eca476cc&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 260 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:01.785932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:01.910496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c48da-4e4d-4ec5-b109-f3402fab6a42 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:01.929329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:01.954741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:01.955421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:01.955694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:01.955960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:01.968533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:01.974004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:01.976200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:01.978131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:01.980609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 187/383] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:01.985425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:02.017248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:02.126404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8273ef4-77c5-40bf-aa90-3600e249206b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:02.179743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:02.180272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:02.180552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:02.180839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:02.181537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:02.181833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:02.183277 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:02.192561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 197/384] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:19:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:02.198227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:02.315945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8553b39-141a-4cb0-ac35-065f14b1f0c8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:02.358510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:02.359191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:02.359465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:02.359735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:02.360837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:02.362511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:02.364518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 188/385] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:02 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:02.369662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:02.476663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:02.496943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58df61b5-c9e9-416c-bf3a-476faf6842ee None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:02.540559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a612381-d875-45f6-a514-139f00a2be64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:02.541197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a612381-d875-45f6-a514-139f00a2be64 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:02.541481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a612381-d875-45f6-a514-139f00a2be64 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:02.541755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a612381-d875-45f6-a514-139f00a2be64 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:02.542933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a612381-d875-45f6-a514-139f00a2be64 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:02.544938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a612381-d875-45f6-a514-139f00a2be64 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:02.546889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 198/386] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:02 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:02.552317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:02.621495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:02.695805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e83b916-16ad-41e6-8046-40b0a50e44a2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:02.752653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd703f6-21fb-4e2c-80db-4eacf876fbd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:02.753242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd703f6-21fb-4e2c-80db-4eacf876fbd5 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:02.753515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd703f6-21fb-4e2c-80db-4eacf876fbd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:02.753786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd703f6-21fb-4e2c-80db-4eacf876fbd5 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:02.755689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd703f6-21fb-4e2c-80db-4eacf876fbd5 None admin] RBAC: Authorizing `identity:create_grant(project_id=55dd5dfdb0894fdfad4138219ed4e9e8, group_id=9e63b8e3c69a4334ae2fc711ca2c36b2, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:02.757110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd703f6-21fb-4e2c-80db-4eacf876fbd5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:02.785725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 189/387] 10.222.0.33 () {58 vars in 1442 bytes} [Thu Oct 6 04:19:02 2022] PUT /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:02.920451 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274a8fdc-f61a-462a-91da-ebfab8ae69ba None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:02.921141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274a8fdc-f61a-462a-91da-ebfab8ae69ba None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:02.921647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274a8fdc-f61a-462a-91da-ebfab8ae69ba None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:02.923265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 199/388] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:02.929181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d9cc2-04ca-4701-8bef-4462bf0ebe13 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:02.932354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d9cc2-04ca-4701-8bef-4462bf0ebe13 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:02.932354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d9cc2-04ca-4701-8bef-4462bf0ebe13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:02.960094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-568d9cc2-04ca-4701-8bef-4462bf0ebe13 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:02.961566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-568d9cc2-04ca-4701-8bef-4462bf0ebe13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:03.061888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 190/389] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:03.376618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274a8fdc-f61a-462a-91da-ebfab8ae69ba None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:03.382385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03aa9fb-3922-46f3-aec8-9a8c68e848c5 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:03.382385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03aa9fb-3922-46f3-aec8-9a8c68e848c5 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:03.382385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03aa9fb-3922-46f3-aec8-9a8c68e848c5 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:03.382385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 191/390] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:03.388473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1002ab0-4064-4b27-82ea-e881d275574d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:03.388997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1002ab0-4064-4b27-82ea-e881d275574d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:03.389488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1002ab0-4064-4b27-82ea-e881d275574d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:03.408900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a1002ab0-4064-4b27-82ea-e881d275574d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:03.410474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a1002ab0-4064-4b27-82ea-e881d275574d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:03.442303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:03.484344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 192/391] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:03.560869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274a8fdc-f61a-462a-91da-ebfab8ae69ba None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:03.613064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1f3315-d71f-442c-b419-43717beccbe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:03.613707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1f3315-d71f-442c-b419-43717beccbe0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:03.613981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1f3315-d71f-442c-b419-43717beccbe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:03.614250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1f3315-d71f-442c-b419-43717beccbe0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:03.630902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1f3315-d71f-442c-b419-43717beccbe0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ed1f3315-d71f-442c-b419-43717beccbe0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:03.636137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:03.637018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:03.638068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 200/392] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:03.643475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1002ab0-4064-4b27-82ea-e881d275574d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:03.748275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46299c-61f9-450b-9492-18f4c7f54f52 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:03.748927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46299c-61f9-450b-9492-18f4c7f54f52 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:03.749493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46299c-61f9-450b-9492-18f4c7f54f52 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:03.751387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 201/393] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:03.756880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab901ed-3ae1-4cb2-bd46-62864ebcc504 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:03.757389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab901ed-3ae1-4cb2-bd46-62864ebcc504 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:03.757881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab901ed-3ae1-4cb2-bd46-62864ebcc504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:03.766991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1002ab0-4064-4b27-82ea-e881d275574d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:03.801694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:03.802532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:03.802981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:03.803192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1ab901ed-3ae1-4cb2-bd46-62864ebcc504 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:03.803600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:03.804410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:03.804524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1ab901ed-3ae1-4cb2-bd46-62864ebcc504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:03.804958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:03.806448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:03.815964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 193/394] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:03.821601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:03.825316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:03.881659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:03.917326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 202/395] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:03.919200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab901ed-3ae1-4cb2-bd46-62864ebcc504 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:03.995595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc4c07-3538-4158-bd90-299a809365e3 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.053475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.054105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.054381 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.054653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.055940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:04.057352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:04.059033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 194/396] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:03 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:04.067271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.085658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab901ed-3ae1-4cb2-bd46-62864ebcc504 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.166763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.167345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.167620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.167887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.172310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:04.181937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f8629-1afa-4cc7-b38f-b4cce6a90a12 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.197195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:04.202663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:04.203619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:04.204515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 203/397] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:04.211345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.216552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:04.248210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.248838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.249163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.249437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.250762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:04.252723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:04.255010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 195/398] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:04.260499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.355868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d516ba-f6b7-45d5-8384-097e50f3dd8e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.382937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cf7bc4-475e-42af-b67f-2e7e99278981 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.404938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.406798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.407306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.408024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.421908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:04.424822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.425503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.425872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.426216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.426950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:04.427614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:04.428982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:04.429514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:04.430394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:04.431271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:04.432775 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 204/399] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:04.432775 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.454246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 196/400] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:04.454246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.483457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:04.528213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:04.572899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1465c71-b687-4c33-b4b8-76a5b9bd6e70 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.580156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9a4de4-64ff-4f7c-a83b-307a93bb7d1c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.616083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.616709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.616987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.617257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.618272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] RBAC: Adding query filter params (name=member) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:04.618583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:04.621092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:04.629019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.630086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.630579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.631103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.635164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 197/401] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:04.637795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.639837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] RBAC: Authorizing `identity:create_grant(project_id=5698e9901f634a3695852e6b8b460d47, group_id=9e63b8e3c69a4334ae2fc711ca2c36b2, role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:04.641323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:04.662985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 205/402] 10.222.0.33 () {58 vars in 1442 bytes} [Thu Oct 6 04:19:04 2022] PUT /identity/v3/projects/5698e9901f634a3695852e6b8b460d47/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:04.662985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.778753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e30510-d091-4b56-af17-dd25b62a3c3b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.779875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:04.796804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2112d7bc-1a4c-4aab-a0fa-6241c601f64d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:04.806450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:04.819239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb962b-e272-420c-bf03-05c43416aa1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.820372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb962b-e272-420c-bf03-05c43416aa1c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.820976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb962b-e272-420c-bf03-05c43416aa1c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.821643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb962b-e272-420c-bf03-05c43416aa1c None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.822938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb962b-e272-420c-bf03-05c43416aa1c None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:04.825191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb962b-e272-420c-bf03-05c43416aa1c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:04.831985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 206/403] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:04.831985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5584be1c-aa6b-447a-8569-7d0b5bcc0431 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.831985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5584be1c-aa6b-447a-8569-7d0b5bcc0431 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.831985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5584be1c-aa6b-447a-8569-7d0b5bcc0431 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.831985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 207/404] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:04.833381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5584be1c-aa6b-447a-8569-7d0b5bcc0431 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.851065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301930f3-df6c-46a3-8d2e-2626900e6beb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:04.852137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301930f3-df6c-46a3-8d2e-2626900e6beb None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.852427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301930f3-df6c-46a3-8d2e-2626900e6beb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.852702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301930f3-df6c-46a3-8d2e-2626900e6beb None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.854213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301930f3-df6c-46a3-8d2e-2626900e6beb None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:04.856664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301930f3-df6c-46a3-8d2e-2626900e6beb None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:04.864754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 198/405] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:04.866643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e02d1-a3cd-40d0-887a-e3c9dcddd653 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:04.866933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e02d1-a3cd-40d0-887a-e3c9dcddd653 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:04.867236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e02d1-a3cd-40d0-887a-e3c9dcddd653 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:04.890353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-860e02d1-a3cd-40d0-887a-e3c9dcddd653 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:04.892517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-860e02d1-a3cd-40d0-887a-e3c9dcddd653 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:04.971695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 199/406] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:04.972593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e02d1-a3cd-40d0-887a-e3c9dcddd653 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:04.986566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5584be1c-aa6b-447a-8569-7d0b5bcc0431 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:05.031374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:05.032078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.032363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.032639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.033845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:05.035463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:05.037767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 208/407] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:05.042949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:05.120232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:05.141030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e02d1-a3cd-40d0-887a-e3c9dcddd653 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:05.180304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:05.180939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.181217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.181490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.182610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:05.184697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:05.185391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:05.187439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 200/408] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:04 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:05.193717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:05.275954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7718496-c31f-404a-8cb2-5280d600ecb8 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:05.322900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:05.323988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.324275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.324607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.327665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] RBAC: Authorizing `identity:create_grant(project_id=5698e9901f634a3695852e6b8b460d47, group_id=9e63b8e3c69a4334ae2fc711ca2c36b2, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:05.329682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:05.349491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4df798-2d83-4748-8852-0a9689da5a23 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:05.351106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 209/409] 10.222.0.33 () {58 vars in 1442 bytes} [Thu Oct 6 04:19:05 2022] PUT /identity/v3/projects/5698e9901f634a3695852e6b8b460d47/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:19:05.351995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:05.401624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3f6e2-477f-4d8f-9cb2-349424a5bcdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:05.402849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3f6e2-477f-4d8f-9cb2-349424a5bcdb None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.403170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3f6e2-477f-4d8f-9cb2-349424a5bcdb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.403506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3f6e2-477f-4d8f-9cb2-349424a5bcdb None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.406901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3f6e2-477f-4d8f-9cb2-349424a5bcdb None admin] RBAC: Authorizing `identity:create_grant(project_id=55dd5dfdb0894fdfad4138219ed4e9e8, group_id=9e63b8e3c69a4334ae2fc711ca2c36b2, role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:05.408746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3f6e2-477f-4d8f-9cb2-349424a5bcdb None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:05.442572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 201/410] 10.222.0.33 () {58 vars in 1442 bytes} [Thu Oct 6 04:19:05 2022] PUT /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8/groups/9e63b8e3c69a4334ae2fc711ca2c36b2/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:19:05.479810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be72839-2ab4-4e1d-bbcc-909a9f33f17d None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.480133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be72839-2ab4-4e1d-bbcc-909a9f33f17d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.480371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be72839-2ab4-4e1d-bbcc-909a9f33f17d None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.483596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 202/411] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:19:05.488303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e080aa7-ff4d-4db8-8635-e62aeaae6967 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.488690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e080aa7-ff4d-4db8-8635-e62aeaae6967 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.489059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e080aa7-ff4d-4db8-8635-e62aeaae6967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.502222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1b1dad-1ec3-43ac-a99e-bfb714d06e96 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:05.516972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6e080aa7-ff4d-4db8-8635-e62aeaae6967 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:05.521110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6e080aa7-ff4d-4db8-8635-e62aeaae6967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:05.539740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:05.546418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:05.583812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:05.584386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.584666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.584933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.612704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:05.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:05.620604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:05.622607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 210/412] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:05.628292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:05.639236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 203/413] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:05.757947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c46874f-2b7d-4672-8747-7563c6effdd8 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:05.794746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:05.795401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.796060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.796349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.797117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:05.797428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:05.798807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:05.807914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 211/414] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:05.814167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e080aa7-ff4d-4db8-8635-e62aeaae6967 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:05.931072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:05.943286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e080aa7-ff4d-4db8-8635-e62aeaae6967 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:05.977560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:05.978185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:05.978457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:05.978728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] PATH_INFO: `/v3/groups/f28434a0fc2e44c78ff390df07003ed5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:05.979854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] RBAC: Authorizing `identity:get_group(group_id=f28434a0fc2e44c78ff390df07003ed5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:05.981375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:05.983255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 204/415] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:05 2022] GET /identity/v3/groups/f28434a0fc2e44c78ff390df07003ed5 => generated 229 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:05.996010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:06.016191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:06.025307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:06.075699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81e974-c5bc-4f27-b3b7-8cef9b7f50c2 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:06.132986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:06.133821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:06.134103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:06.134378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:06.157477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:06.162989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:06.163868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:06.164674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:06.169995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 212/416] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:06.176062 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:06.208027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7811fd-9922-433d-af8e-4749d803489e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:06.265225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:06.265863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:06.266132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:06.266399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] PATH_INFO: `/v3/projects/7760d317949542fd9d86e529eca476cc` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:06.267562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] RBAC: Authorizing `identity:get_project(project_id=7760d317949542fd9d86e529eca476cc)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:06.269224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:06.270871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 205/417] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:05 2022] GET /identity/v3/projects/7760d317949542fd9d86e529eca476cc => generated 344 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:06.278721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:06.369509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:06.374043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef73f7c8-edaa-49db-b0ca-44c8a79a5aab None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:06.411399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:06.429828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:06.430543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:06.430820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:06.431131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:06.434996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] RBAC: Adding query filter params (name=member) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:06.434996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:06.434996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:06.442856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 213/418] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:19:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:06.448483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:06.451736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6ae167-a99a-43d1-be6c-d8c619188cd0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:06.509765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:06.510349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:06.510703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:06.510987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] PATH_INFO: `/v3/projects/7760d317949542fd9d86e529eca476cc/groups/f28434a0fc2e44c78ff390df07003ed5/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:06.513121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] RBAC: Authorizing `identity:create_grant(project_id=7760d317949542fd9d86e529eca476cc, group_id=f28434a0fc2e44c78ff390df07003ed5, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:06.514532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:06.532129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 206/419] 10.222.0.33 () {58 vars in 1442 bytes} [Thu Oct 6 04:19:06 2022] PUT /identity/v3/projects/7760d317949542fd9d86e529eca476cc/groups/f28434a0fc2e44c78ff390df07003ed5/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:06.584705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a05064-7886-415e-bfba-7b3681436866 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:06.630951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:06.631662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:06.631997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:06.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:06.665965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:06.714593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:06.716618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:06.844216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 214/420] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 06 04:19:06.849810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:06.957412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5913b48-b60b-41ca-ae62-c62cca7db109 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:06.993783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:06.994408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:06.994685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:06.994949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:06.996139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:06.998425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:07.000423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 207/421] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:06 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:07.005918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:07.070087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:07.114876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554e638e-d4d5-4b1c-91ec-f9db18607e9a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:07.172688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca88192-a2ab-4a74-a4d2-8103ae33500d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:07.173343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca88192-a2ab-4a74-a4d2-8103ae33500d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:07.173619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca88192-a2ab-4a74-a4d2-8103ae33500d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:07.173887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca88192-a2ab-4a74-a4d2-8103ae33500d None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:07.175027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca88192-a2ab-4a74-a4d2-8103ae33500d None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:07.176450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca88192-a2ab-4a74-a4d2-8103ae33500d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:07.178234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 215/422] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:07 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:07.184787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:07.196082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:07.266582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7de64b-7597-40a4-a137-647e4a44fa49 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:07.266862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7de64b-7597-40a4-a137-647e4a44fa49 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:07.267164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7de64b-7597-40a4-a137-647e4a44fa49 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:07.268478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 216/423] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:07.273487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d97088f-951e-472c-9a8a-b4cf26b33a87 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:07.273758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d97088f-951e-472c-9a8a-b4cf26b33a87 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:07.274024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d97088f-951e-472c-9a8a-b4cf26b33a87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:07.304278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-4d97088f-951e-472c-9a8a-b4cf26b33a87 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:07.306817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4d97088f-951e-472c-9a8a-b4cf26b33a87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:07.330507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b861d7ba-b6cc-4364-b038-758a79c2080e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:07.397484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:07.398021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:07.398286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:07.398550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:07.399354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:07.399667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:07.401217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:07.415934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 208/424] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:07 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 611 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:07.419007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 217/425] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:07.755494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:07.867836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:07.913338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb49c1d-222b-46c4-9cfe-daf9c3f9d073 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:07.957742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26747414-5409-4bed-9736-106916a089d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:07.958383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26747414-5409-4bed-9736-106916a089d8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:07.958658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26747414-5409-4bed-9736-106916a089d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:07.958930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26747414-5409-4bed-9736-106916a089d8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:07.972823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26747414-5409-4bed-9736-106916a089d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-26747414-5409-4bed-9736-106916a089d8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:07.978397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:07.979424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:07.980238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:07.983061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 209/426] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:07.986477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d97088f-951e-472c-9a8a-b4cf26b33a87 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:07.987359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb524b54-41ae-4c48-8ca6-aa63e56aaf91 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:07.987537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb524b54-41ae-4c48-8ca6-aa63e56aaf91 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:07.987717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb524b54-41ae-4c48-8ca6-aa63e56aaf91 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:07.989367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 210/427] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:07.998414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8c3158-87af-4bad-8866-d6eb67216595 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:07.998939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8c3158-87af-4bad-8866-d6eb67216595 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:07.999447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8c3158-87af-4bad-8866-d6eb67216595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.024106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4b8c3158-87af-4bad-8866-d6eb67216595 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:08.025358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4b8c3158-87af-4bad-8866-d6eb67216595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:08.068622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:08.120870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 211/428] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:08.126095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d97088f-951e-472c-9a8a-b4cf26b33a87 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:08.163853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:08.164463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.164744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.165013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.165680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:08.165982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:08.167507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:08.180726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 218/429] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:08.182091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8c3158-87af-4bad-8866-d6eb67216595 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:08.284996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:08.329239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8c3158-87af-4bad-8866-d6eb67216595 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:08.368059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7d22b6-449d-4db6-abd4-dbacf1ea82c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:08.369013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d22b6-449d-4db6-abd4-dbacf1ea82c2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.369458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d22b6-449d-4db6-abd4-dbacf1ea82c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.369969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7d22b6-449d-4db6-abd4-dbacf1ea82c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.440229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7d22b6-449d-4db6-abd4-dbacf1ea82c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:08.441985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7d22b6-449d-4db6-abd4-dbacf1ea82c2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:08.452814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:08.511322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:08.565092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 212/430] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:08.565092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59db73ae-e1af-42a0-8577-329f77cb52c3 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.565092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59db73ae-e1af-42a0-8577-329f77cb52c3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.565092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59db73ae-e1af-42a0-8577-329f77cb52c3 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.565092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 213/431] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:08 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:08.567283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59db73ae-e1af-42a0-8577-329f77cb52c3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:08.590705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bf5b8e-733a-4665-8c55-448ef75f2ac5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:08.620790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:08.637295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87351c39-30f9-4169-836d-e6f42cc4dddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:08.638069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87351c39-30f9-4169-836d-e6f42cc4dddf None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.638343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87351c39-30f9-4169-836d-e6f42cc4dddf None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.638607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87351c39-30f9-4169-836d-e6f42cc4dddf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.649562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87351c39-30f9-4169-836d-e6f42cc4dddf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-87351c39-30f9-4169-836d-e6f42cc4dddf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:08.656601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:08.657663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 06 04:19:08.658477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:08.659636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 219/432] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:08.661359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990474a-2723-4023-b96b-9d116806e2f1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.661638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990474a-2723-4023-b96b-9d116806e2f1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.661903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990474a-2723-4023-b96b-9d116806e2f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.680471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3990474a-2723-4023-b96b-9d116806e2f1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:08.681567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3990474a-2723-4023-b96b-9d116806e2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:08.687191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59db73ae-e1af-42a0-8577-329f77cb52c3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:08.725178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:08.725801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.726075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.726343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.727497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:08.728907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:08.730556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 214/433] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:08 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:08.733543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:08.759933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 220/434] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:08.761085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3990474a-2723-4023-b96b-9d116806e2f1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:08.781536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:08.874656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3990474a-2723-4023-b96b-9d116806e2f1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:08.875954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aefe5fb-227c-464b-a047-a48b00a74adf None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:08.892908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:08.922940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:08.923392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:08.923889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.924271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.924442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:08.924730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.925054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:08.925561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:08.926184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:08.926556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] RBAC: Adding query filter params (name=member) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:08.927039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:08.928007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:08.928702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:08.939324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 221/435] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:08 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:08.941034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:08.951713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 215/436] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:19:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:08.957496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.052506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.067599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1249ec-e76a-4b59-ab8b-11838fa7b308 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.078892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a8eb87-f4c4-4884-9b52-f44b6758178b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.104276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27eb32a-f398-400f-8209-8f2140aabf74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.104875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27eb32a-f398-400f-8209-8f2140aabf74 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.105149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27eb32a-f398-400f-8209-8f2140aabf74 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.105416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27eb32a-f398-400f-8209-8f2140aabf74 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.114788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.115424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.115699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.115972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.116798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=5698e9901f634a3695852e6b8b460d47) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:09.117109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:09.118535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:09.125247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 222/437] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:08 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 611 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:09.125861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.181040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.216635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27eb32a-f398-400f-8209-8f2140aabf74 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:09.217681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27eb32a-f398-400f-8209-8f2140aabf74 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:09.258284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24857148-a477-4973-8e28-077cb9136e72 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.301641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.302323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.302603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.302875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.315086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:09.320445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:09.321353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:09.322148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 223/438] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:09.329132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.366354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.378052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 216/439] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 06 04:19:09.379352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c46e4b-7e5f-4108-8e16-869c6faf5427 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.379631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c46e4b-7e5f-4108-8e16-869c6faf5427 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.379949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c46e4b-7e5f-4108-8e16-869c6faf5427 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.381319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 217/440] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:09.383994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c46e4b-7e5f-4108-8e16-869c6faf5427 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.441831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56574aa-3ffe-4532-a0d8-1b3a88059ff0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.444203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.476883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1ce525-b6ae-4add-8f1b-4b94044fb35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.477474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1ce525-b6ae-4add-8f1b-4b94044fb35b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.477753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1ce525-b6ae-4add-8f1b-4b94044fb35b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.478018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1ce525-b6ae-4add-8f1b-4b94044fb35b None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.478661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1ce525-b6ae-4add-8f1b-4b94044fb35b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:09.478976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1ce525-b6ae-4add-8f1b-4b94044fb35b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:09.480369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1ce525-b6ae-4add-8f1b-4b94044fb35b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:09.488294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c46e4b-7e5f-4108-8e16-869c6faf5427 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.493084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 224/441] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:09.493084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b059edd1-e3c1-4aec-b49d-5d78b82422f3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.493084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b059edd1-e3c1-4aec-b49d-5d78b82422f3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.493084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b059edd1-e3c1-4aec-b49d-5d78b82422f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.508030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b059edd1-e3c1-4aec-b49d-5d78b82422f3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:09.509207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b059edd1-e3c1-4aec-b49d-5d78b82422f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:09.525931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.526551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.526819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.527119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] PATH_INFO: `/v3/projects/5698e9901f634a3695852e6b8b460d47` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.528916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] RBAC: Authorizing `identity:get_project(project_id=5698e9901f634a3695852e6b8b460d47)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:09.530318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:09.531946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 218/442] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/projects/5698e9901f634a3695852e6b8b460d47 => generated 302 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:09.532683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.588561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 225/443] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:09.591824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b059edd1-e3c1-4aec-b49d-5d78b82422f3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.594375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.644020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93afa2f3-fd79-4784-8073-27f32cf6b650 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.686666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.687299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.687576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.687844 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.698153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.705785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b059edd1-e3c1-4aec-b49d-5d78b82422f3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.741945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.742574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.742851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.743152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.744258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:09.745930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:09.748517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 226/444] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:09.753992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.765055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:09.766485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:09.828174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.858601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44499b74-0ee1-4717-a237-89224d7e55df None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:09.875254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 219/445] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:09.876096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.894828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:09.895382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:09.895658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:09.895927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:09.896647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=68ada91d9625461ea199ad5a616e8f01, scope.project.id=5698e9901f634a3695852e6b8b460d47) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:09.896952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:09.898372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:09.904260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 227/446] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=5698e9901f634a3695852e6b8b460d47&role.id=68ada91d9625461ea199ad5a616e8f01 => generated 611 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:09.905025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:09.937895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:09.988777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f11c01-1899-4ca6-a124-1d37e852c7d5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.021834 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e80b01-3ce4-48fd-aefd-985a093ffc64 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.027129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5413b26-5152-4053-8131-082281af7698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.027834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5413b26-5152-4053-8131-082281af7698 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.028115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5413b26-5152-4053-8131-082281af7698 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.028389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5413b26-5152-4053-8131-082281af7698 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.040069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5413b26-5152-4053-8131-082281af7698 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-d5413b26-5152-4053-8131-082281af7698 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:10.045987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:10.047010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:10.047846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 220/447] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:10.053042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5413b26-5152-4053-8131-082281af7698 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:10.064443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.065236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.065521 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.065800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] PATH_INFO: `/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.066984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] RBAC: Authorizing `identity:get_project(project_id=55dd5dfdb0894fdfad4138219ed4e9e8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.068460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:10.070408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 228/448] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:09 2022] GET /identity/v3/projects/55dd5dfdb0894fdfad4138219ed4e9e8 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:10.076108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:10.088653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:10.177560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5413b26-5152-4053-8131-082281af7698 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.181395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:10.187539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1ac5fd-fd9a-4a88-8c90-d2fa1c5d10c8 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.213218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.213813 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.214084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.214355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.215697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:10.216004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.217392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:10.222633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.223298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.223579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.223855 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] PATH_INFO: `/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.224998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] RBAC: Authorizing `identity:get_group(group_id=9e63b8e3c69a4334ae2fc711ca2c36b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.226188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 221/449] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:10.226535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:10.228212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 229/450] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:10 2022] GET /identity/v3/groups/9e63b8e3c69a4334ae2fc711ca2c36b2 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:10.232005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:10.233696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:10.314540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:10.331981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec642802-9cea-4e52-9129-f31cd734d203 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.332302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36315ca-e375-42df-8200-e17aa1deb4d9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.365182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.365631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.365797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.366070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.366222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.366338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.366500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.366771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.367080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] RBAC: Adding query filter params (group.id=9e63b8e3c69a4334ae2fc711ca2c36b2, role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:10.367388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.368843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:10.374548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 230/451] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:10 2022] GET /identity/v3/role_assignments?group.id=9e63b8e3c69a4334ae2fc711ca2c36b2&scope.project.id=55dd5dfdb0894fdfad4138219ed4e9e8&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 611 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:10.438319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.439719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:10.524586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:10.544447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 222/452] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:10.550097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:10.580576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:10.657965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33a748b-2c7b-4563-a2bb-faa1a220ffc2 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.692466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.693019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.693298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.693573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] PATH_INFO: `/v3/projects/7760d317949542fd9d86e529eca476cc` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.694691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] RBAC: Authorizing `identity:get_project(project_id=7760d317949542fd9d86e529eca476cc)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.696199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:10.697823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 231/453] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:19:10 2022] GET /identity/v3/projects/7760d317949542fd9d86e529eca476cc => generated 344 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:10.704058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:10.804915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f721fbd-46b5-4cb7-98dd-161aa56cc2d2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.840772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38da7149-e10f-44e1-9296-9f1c03374d51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.841472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da7149-e10f-44e1-9296-9f1c03374d51 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.841753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da7149-e10f-44e1-9296-9f1c03374d51 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.842022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da7149-e10f-44e1-9296-9f1c03374d51 None admin] PATH_INFO: `/v3/groups/f28434a0fc2e44c78ff390df07003ed5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.843161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38da7149-e10f-44e1-9296-9f1c03374d51 None admin] RBAC: Authorizing `identity:get_group(group_id=f28434a0fc2e44c78ff390df07003ed5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.844582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38da7149-e10f-44e1-9296-9f1c03374d51 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:10.846150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 223/454] 10.222.0.33 () {56 vars in 1180 bytes} [Thu Oct 6 04:19:10 2022] GET /identity/v3/groups/f28434a0fc2e44c78ff390df07003ed5 => generated 229 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:10.852157 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:10.951965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:10.957815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6640b53-5f65-4b51-86e8-445562029ad6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:10.992473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9555cb-e083-4c72-a18f-9e9dc5286400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:10.993068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9555cb-e083-4c72-a18f-9e9dc5286400 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:10.993345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9555cb-e083-4c72-a18f-9e9dc5286400 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:10.993609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9555cb-e083-4c72-a18f-9e9dc5286400 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:10.994322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9555cb-e083-4c72-a18f-9e9dc5286400 None admin] RBAC: Adding query filter params (group.id=f28434a0fc2e44c78ff390df07003ed5, role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=7760d317949542fd9d86e529eca476cc) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:10.994620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9555cb-e083-4c72-a18f-9e9dc5286400 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:10.996087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9555cb-e083-4c72-a18f-9e9dc5286400 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:11.001922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 232/455] 10.222.0.33 () {56 vars in 1376 bytes} [Thu Oct 6 04:19:10 2022] GET /identity/v3/role_assignments?group.id=f28434a0fc2e44c78ff390df07003ed5&scope.project.id=7760d317949542fd9d86e529eca476cc&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:13.556468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3dcc15-0288-4f9b-a947-cbd25e4e74fb None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:13.556974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3dcc15-0288-4f9b-a947-cbd25e4e74fb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:13.556974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3dcc15-0288-4f9b-a947-cbd25e4e74fb None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:13.559060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 224/456] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:13.564650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1404309-6d98-4414-91d3-daf7c8eb79a0 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:13.565154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1404309-6d98-4414-91d3-daf7c8eb79a0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:13.565256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1404309-6d98-4414-91d3-daf7c8eb79a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:13.591985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f1404309-6d98-4414-91d3-daf7c8eb79a0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:13.593213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f1404309-6d98-4414-91d3-daf7c8eb79a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:13.679605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 233/457] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:14.007910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec22a849-88be-48bf-b208-a88ac10ed156 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.008191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec22a849-88be-48bf-b208-a88ac10ed156 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.008621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec22a849-88be-48bf-b208-a88ac10ed156 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.010358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 225/458] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:14.016216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496985ee-d200-422f-a849-f24155d03535 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.016684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496985ee-d200-422f-a849-f24155d03535 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.017109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496985ee-d200-422f-a849-f24155d03535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.041492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-496985ee-d200-422f-a849-f24155d03535 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:14.041823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec22a849-88be-48bf-b208-a88ac10ed156 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:14.043101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-496985ee-d200-422f-a849-f24155d03535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:14.120769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:14.134216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 234/459] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:14.135788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8163b5-083d-4023-9106-15c3b940d794 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.136211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8163b5-083d-4023-9106-15c3b940d794 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.136667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8163b5-083d-4023-9106-15c3b940d794 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.138418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 235/460] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:14.144213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de32c3c-eea3-4478-9651-9b792df5e69d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.144647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de32c3c-eea3-4478-9651-9b792df5e69d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.145081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de32c3c-eea3-4478-9651-9b792df5e69d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.166516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5de32c3c-eea3-4478-9651-9b792df5e69d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:14.168045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5de32c3c-eea3-4478-9651-9b792df5e69d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:14.186643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:14.234981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec22a849-88be-48bf-b208-a88ac10ed156 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:14.286426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 236/461] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:14.298000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:14.298676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.298966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.299239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.310502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:14.314823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:14.327719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:14.329820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:14.331377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:14.331377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:14.331377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:14.331377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 226/462] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:14.335028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de32c3c-eea3-4478-9651-9b792df5e69d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:14.484239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:14.498841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de32c3c-eea3-4478-9651-9b792df5e69d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:14.583022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:14.583790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.583943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.584163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.584816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:14.585113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:14.586482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:14.607659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d2908-f8a5-4fde-8156-79e34b286ea0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:14.611034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:14.618694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 237/463] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:14.624218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:14.637694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:14.661783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf081db-272b-4398-bff1-fa3dab50afe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:14.662731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf081db-272b-4398-bff1-fa3dab50afe3 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.663244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf081db-272b-4398-bff1-fa3dab50afe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.663713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf081db-272b-4398-bff1-fa3dab50afe3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.682049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf081db-272b-4398-bff1-fa3dab50afe3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:14.684078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf081db-272b-4398-bff1-fa3dab50afe3 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-ebf081db-272b-4398-bff1-fa3dab50afe3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:14.703581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:14.704550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:14.705415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:14.705415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:14.705415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:14.707035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 227/464] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:14.708434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af35b37-9152-43cf-ba29-aa88b2c742f1 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.708921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af35b37-9152-43cf-ba29-aa88b2c742f1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.709383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af35b37-9152-43cf-ba29-aa88b2c742f1 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.711124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 228/465] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:14.712117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af35b37-9152-43cf-ba29-aa88b2c742f1 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:14.747650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6778ea-7356-42da-b791-9146cc8703db None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:14.784091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668a610f-d034-481e-926d-1a63ef7334f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:14.784713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668a610f-d034-481e-926d-1a63ef7334f1 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.784993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668a610f-d034-481e-926d-1a63ef7334f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.785268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668a610f-d034-481e-926d-1a63ef7334f1 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.785915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668a610f-d034-481e-926d-1a63ef7334f1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:14.787378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668a610f-d034-481e-926d-1a63ef7334f1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:14.810412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 238/466] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:19:14 2022] POST /identity/v3/users => generated 246 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:19:14.810412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e5c18-7af0-4641-9b2a-41a6a9fcee24 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.810412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e5c18-7af0-4641-9b2a-41a6a9fcee24 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.810412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e5c18-7af0-4641-9b2a-41a6a9fcee24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.836762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-035e5c18-7af0-4641-9b2a-41a6a9fcee24 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:14.837906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-035e5c18-7af0-4641-9b2a-41a6a9fcee24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:14.839264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af35b37-9152-43cf-ba29-aa88b2c742f1 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:14.901011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:14.902009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:14.902498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:14.902907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:14.904067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:14.904601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:14.907227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:14.921978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:14.927642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 229/467] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:14.928824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:14.937452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 239/468] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:14.938454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035e5c18-7af0-4641-9b2a-41a6a9fcee24 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:14.951951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:15.063579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d248974-74bf-4ea8-b7f8-aa5ff62e6ddf None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.075217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035e5c18-7af0-4641-9b2a-41a6a9fcee24 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.112510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.113207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.113484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.113810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.117981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b901aed-36db-4592-935f-2e371cd9d3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.118500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b901aed-36db-4592-935f-2e371cd9d3ae None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.118686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b901aed-36db-4592-935f-2e371cd9d3ae None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.118917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b901aed-36db-4592-935f-2e371cd9d3ae None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.120033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b901aed-36db-4592-935f-2e371cd9d3ae None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.121600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b901aed-36db-4592-935f-2e371cd9d3ae None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:15.126069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.127541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:15.144643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 240/469] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:19:14 2022] POST /identity/v3/users => generated 248 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:15.145904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e12a2-536d-4ad9-b078-bd326a66e479 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.146190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e12a2-536d-4ad9-b078-bd326a66e479 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.146464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e12a2-536d-4ad9-b078-bd326a66e479 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:15.146753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:15.147798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:15.148684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:15.148684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:15.148684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:15.148849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 241/470] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:15.150604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 230/471] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:15.161115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:15.161567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37c88e-32d0-4695-897a-83d52e35c753 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.161920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37c88e-32d0-4695-897a-83d52e35c753 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.162262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e37c88e-32d0-4695-897a-83d52e35c753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.189220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6e37c88e-32d0-4695-897a-83d52e35c753 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:15.190429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6e37c88e-32d0-4695-897a-83d52e35c753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:15.287974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:15.293512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:15.315158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 242/472] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:15.319249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e37c88e-32d0-4695-897a-83d52e35c753 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:15.366270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032b557c-30e9-4151-ba65-6bb632f5ae2a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.415817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.418157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.418157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.418157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.418157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:15.418157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.419547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:15.435549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 231/473] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:15.440906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:15.460259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e37c88e-32d0-4695-897a-83d52e35c753 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.505583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.506210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.506520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.506789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.519775 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.521705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:15.540475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:15.541658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:15.542777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:15.542777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:15.542777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:15.544891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 243/474] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:15.555992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:15.572417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bd7274-aa19-498c-abf1-f65025f363f0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.594434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:15.605467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:15.614374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.615194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.615452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.615718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.616544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.617910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:15.645084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 232/475] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:19:15 2022] POST /identity/v3/users => generated 248 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:15.645999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:15.682235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086b0396-6c2c-40fd-ad98-14297e4ddc4f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.725386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.725386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.725386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.725386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.725970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:15.726184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.727800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:15.749596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 244/476] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:15.751626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:15.771051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76a2766-2140-4901-8cb7-ce449abe9603 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.812118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.813144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.813642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.814109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.815161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.816959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:15.831093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.sql.core [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 06 04:19:15.831093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 06 04:19:15.831093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [parameters: {'id': 'b7783a43ee714654a3ebf8b3d8164f90', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 06 04:19:15.831093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81399) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application result = self._query(query) Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application conn.query(q) Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application result.read() Oct 06 04:19:15.857488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return ref.to_dict() Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 06 04:19:15.858442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application next(self.gen) Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application yield resource Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application next(self.gen) Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self.session.rollback() Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self.force_reraise() Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise self.value Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application session.commit() Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 06 04:19:15.859396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._prepare_impl() Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self.session.flush() Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._flush(objects) Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application compat.raise_( Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application flush_context.execute() Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rec.execute(self) Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 06 04:19:15.860322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 06 04:19:15.861213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application result = self._query(query) Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application conn.query(q) Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application result.read() Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 06 04:19:15.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [parameters: {'id': 'b7783a43ee714654a3ebf8b3d8164f90', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:15.864390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 06 04:19:15.866369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:15.870197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 233/477] 10.222.0.33 () {60 vars in 1131 bytes} [Thu Oct 6 04:19:15 2022] POST /identity/v3/roles => generated 144 bytes in 224 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:15.876924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:15.909178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:15.924339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383a0094-339f-44c3-9002-90c5bc2820cd None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:15.962416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:15.991423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:15.991755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:15.991922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:15.992191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:15.992841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:15.994269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:16.031009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 245/478] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:19:15 2022] POST /identity/v3/users => generated 249 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:16.034231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d633ab1-4300-4144-8dfe-f875314c59e6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:16.075831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0b6606-0518-439d-b272-1e295cfa2c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:16.076467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0b6606-0518-439d-b272-1e295cfa2c72 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:16.076784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0b6606-0518-439d-b272-1e295cfa2c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:16.077056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0b6606-0518-439d-b272-1e295cfa2c72 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:16.095777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0b6606-0518-439d-b272-1e295cfa2c72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-df0b6606-0518-439d-b272-1e295cfa2c72 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:16.103407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:16.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:16.106148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 234/479] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:16.112105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:16.274332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9ab1fa-008b-4dbe-a652-1980c5a8196e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:16.317096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bdc5a3-dff8-4af0-9343-fb6a183247de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:16.317763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdc5a3-dff8-4af0-9343-fb6a183247de None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:16.318041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdc5a3-dff8-4af0-9343-fb6a183247de None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:16.318309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bdc5a3-dff8-4af0-9343-fb6a183247de None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:16.319001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bdc5a3-dff8-4af0-9343-fb6a183247de None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:16.319305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bdc5a3-dff8-4af0-9343-fb6a183247de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:16.320706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bdc5a3-dff8-4af0-9343-fb6a183247de None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:16.331163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 246/480] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:17.475324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944db29-aec8-4558-9410-5a85d605fcaf None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:17.475833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944db29-aec8-4558-9410-5a85d605fcaf None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:17.475833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944db29-aec8-4558-9410-5a85d605fcaf None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:17.477316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 235/481] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:17.483457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59020ba-8966-4f0e-894e-3cbf824dd5e2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:17.483858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59020ba-8966-4f0e-894e-3cbf824dd5e2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:17.484048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59020ba-8966-4f0e-894e-3cbf824dd5e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:17.509176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d59020ba-8966-4f0e-894e-3cbf824dd5e2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:17.510348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d59020ba-8966-4f0e-894e-3cbf824dd5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:17.518427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:17.581375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 247/482] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:17.895256 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7944db29-aec8-4558-9410-5a85d605fcaf None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:18.023736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7944db29-aec8-4558-9410-5a85d605fcaf None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:18.032540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:18.064415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab5ab58-2c06-4e25-add2-c5167b65f629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:18.065055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5ab58-2c06-4e25-add2-c5167b65f629 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.065328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5ab58-2c06-4e25-add2-c5167b65f629 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.065595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5ab58-2c06-4e25-add2-c5167b65f629 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.077352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab5ab58-2c06-4e25-add2-c5167b65f629 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7ab5ab58-2c06-4e25-add2-c5167b65f629 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:18.082310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:18.083254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:18.084303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 236/483] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:18.092213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59020ba-8966-4f0e-894e-3cbf824dd5e2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:18.176802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a39438-4819-4cca-a0e1-6441073ea7e0 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.177158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a39438-4819-4cca-a0e1-6441073ea7e0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.177537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a39438-4819-4cca-a0e1-6441073ea7e0 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.179239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 237/484] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:19:18.184512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c5acea-2c7e-44cf-937f-4928f1a59c76 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.184898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c5acea-2c7e-44cf-937f-4928f1a59c76 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.185272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c5acea-2c7e-44cf-937f-4928f1a59c76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.206808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c1c5acea-2c7e-44cf-937f-4928f1a59c76 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:18.208227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c1c5acea-2c7e-44cf-937f-4928f1a59c76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:18.279942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59020ba-8966-4f0e-894e-3cbf824dd5e2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:18.338517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 238/485] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:18.343748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:18.344531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.344981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.345398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.346211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:18.346657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:18.348271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:18.357878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 248/486] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:18.368102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c5acea-2c7e-44cf-937f-4928f1a59c76 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:18.493970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:18.504086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c5acea-2c7e-44cf-937f-4928f1a59c76 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:18.549072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:18.549978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.550510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.550985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.561960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:18.563638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:18.578912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:18.579885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:18.580776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:18.580776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:18.580776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:18.582612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 239/487] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:18.587862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:18.603370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:18.699527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2369751-331a-40b4-9dee-b2b5df38b03c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:18.718130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:18.766305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:18.766878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.768014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.768300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.768983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:18.769288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:18.770723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:18.798051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 249/488] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:18.803713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:18.879462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:18.880154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab680a9-828c-4d25-a5a9-5a147e0f724f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:18.916606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109c00a3-d89a-4ad8-9bfc-a560bdf78dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:18.917227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109c00a3-d89a-4ad8-9bfc-a560bdf78dd4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.917505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109c00a3-d89a-4ad8-9bfc-a560bdf78dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.917814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109c00a3-d89a-4ad8-9bfc-a560bdf78dd4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.926267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500d2cbc-71c6-40d7-8518-0569b4900cfb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:18.928356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109c00a3-d89a-4ad8-9bfc-a560bdf78dd4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-109c00a3-d89a-4ad8-9bfc-a560bdf78dd4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:18.933567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:18.934469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:18.936074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 240/489] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:18.936774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e907d-6b02-4593-8edb-52fb6b7aaea3 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.936774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e907d-6b02-4593-8edb-52fb6b7aaea3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.936774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e907d-6b02-4593-8edb-52fb6b7aaea3 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.942116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 241/490] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:18 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:18.943106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06205b1-8b08-4cab-a64e-411d6fae7159 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.943388 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06205b1-8b08-4cab-a64e-411d6fae7159 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.943658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06205b1-8b08-4cab-a64e-411d6fae7159 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.945052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 242/491] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:18 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:18.945515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06205b1-8b08-4cab-a64e-411d6fae7159 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:18.957802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:18.970833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d9caa3-3fac-41d5-8f98-b77329ea6632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:18.971527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d9caa3-3fac-41d5-8f98-b77329ea6632 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:18.971806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d9caa3-3fac-41d5-8f98-b77329ea6632 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:18.972071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d9caa3-3fac-41d5-8f98-b77329ea6632 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:18.980367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d9caa3-3fac-41d5-8f98-b77329ea6632 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:18.982002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d9caa3-3fac-41d5-8f98-b77329ea6632 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-83d9caa3-3fac-41d5-8f98-b77329ea6632 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:18.998935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:19.000003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:19.000941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:19.002253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 250/492] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:19:18 2022] GET /identity/v3/users/nova => generated 82 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:19.003630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9cb2da-ed04-4a3c-b47a-00688dc1cf6b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.003909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9cb2da-ed04-4a3c-b47a-00688dc1cf6b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.004177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9cb2da-ed04-4a3c-b47a-00688dc1cf6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.023298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9e9cb2da-ed04-4a3c-b47a-00688dc1cf6b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:19.024882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9e9cb2da-ed04-4a3c-b47a-00688dc1cf6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:19.069131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06205b1-8b08-4cab-a64e-411d6fae7159 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.108256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 251/493] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:19.108256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11686fe6-35fa-499e-ae4e-a2b29af99ba3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.108256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11686fe6-35fa-499e-ae4e-a2b29af99ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.108256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11686fe6-35fa-499e-ae4e-a2b29af99ba3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.115274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-11686fe6-35fa-499e-ae4e-a2b29af99ba3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:19.116460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-11686fe6-35fa-499e-ae4e-a2b29af99ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:19.120567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.121408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.121759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.122101 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.122820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:19.123214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:19.124643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:19.132911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 243/494] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:19.133772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.172165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:19.192719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 252/495] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:19.194158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7adc6-34da-4a18-a1df-11dc5964194a None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.194435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7adc6-34da-4a18-a1df-11dc5964194a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.194702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7adc6-34da-4a18-a1df-11dc5964194a None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.195950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 253/496] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:19 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:19.196628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7adc6-34da-4a18-a1df-11dc5964194a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.227438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:19.260982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57775a18-ccba-4edc-aca3-027765c42c12 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.296750 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2102d365-d95d-4a86-8da8-e64722513b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.297281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102d365-d95d-4a86-8da8-e64722513b14 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.297626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102d365-d95d-4a86-8da8-e64722513b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.297960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2102d365-d95d-4a86-8da8-e64722513b14 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.298887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2102d365-d95d-4a86-8da8-e64722513b14 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:19.299153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2102d365-d95d-4a86-8da8-e64722513b14 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:19.300639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2102d365-d95d-4a86-8da8-e64722513b14 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:19.334284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7adc6-34da-4a18-a1df-11dc5964194a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.341035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 244/497] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:19.342225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08853239-3f3e-463d-be0e-cf8415fbd02d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.342570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08853239-3f3e-463d-be0e-cf8415fbd02d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.342908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08853239-3f3e-463d-be0e-cf8415fbd02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.360906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-08853239-3f3e-463d-be0e-cf8415fbd02d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:19.362429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-08853239-3f3e-463d-be0e-cf8415fbd02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:19.379104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.379643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.380036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.380443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.391962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:19.393515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:19.410267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:19.411225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:19.412119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:19.412119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:19.412119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:19.412119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 254/498] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:19.412336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.447110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 245/499] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:19.448087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08853239-3f3e-463d-be0e-cf8415fbd02d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.456650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:19.515119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:19.533018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccbe86e-78cc-43b9-92e6-4490705b5b34 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.561787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08853239-3f3e-463d-be0e-cf8415fbd02d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.569275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.570134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.570408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.570680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.602174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.602816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.603133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.603404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.615139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:19.619903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:19.621004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:19.621822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 246/500] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:19.622500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.648286 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:19.649730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:19.685295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:19.729261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4436af-db4f-44c0-a77a-b6ebab96e26a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.748688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:19.751602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 255/501] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:19.752486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.766572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.767194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.767473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.767740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.768379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:19.768676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:19.770039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:19.781121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 247/502] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:19.781896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.854692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9298f-0d54-4056-8056-3db7164eaf34 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.881917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52795ebd-8c5f-48cb-8d22-6f9ff62607fd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:19.889699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.890313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.890623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.890852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:19.901723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:19.906594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:19.907527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:19.908352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 256/503] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:19.909010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.920409 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:19.921006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:19.921281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:19.921552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:19.922209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:19.922518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:19.923905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:19.931762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 248/504] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:19.932549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:19.972214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.012824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf5deac-d3fd-4970-8733-12cfbfdbd805 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.032593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9d4dbb-1d63-468e-be97-d5535bc26115 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.046618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.047259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.047536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.047804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.058105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.059693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.065390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.065999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.066282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.066551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:20.074172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:20.075099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:20.075971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:20.075971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.075971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:20.075971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 257/505] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:20.076368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.079516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.081022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:20.096005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:20.096924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:20.097766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:20.097766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.097766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:20.097766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 249/506] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:20.097999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.125622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.175979 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9e7b20-966f-4755-8d4e-f2d210f415e5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.193035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.203439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e9f6ab-cac7-46c8-81ae-e615b2744ebe None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.209719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.210406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.210677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.210948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.218651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.220176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:20.231495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:20.232417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 258/507] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/users/cinder => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:20.233325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.242165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.242765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.243067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.243340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.243993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:20.244299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.246576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.254687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 250/508] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:20.255501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.333567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a98947b-4745-4f28-a6ec-5a23d0029c23 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.344569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.355242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb12ad9-f5c5-4696-a2a0-c648784aa26e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.369648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.370270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.370549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.370925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.381329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.382730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.389032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.389629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.389913 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.390190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.390820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:20.391156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.392530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:20.397642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:20.398584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:20.399486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:20.399486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.399486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:20.399486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 259/509] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:20.399486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.411144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 251/510] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:20.411144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.424502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.498268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63842e7-559d-4ccb-8210-e4dc5bf31df2 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.508589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeaaba1-aaa3-4b59-959a-584af9fac7ff None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.533323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.533912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.534194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.534458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.535123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:20.535422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.536773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.542343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.543381 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.543816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.544249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.545015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:20.545355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.547272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.548906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 260/511] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:20.549688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.564918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 252/512] 10.222.0.33 () {56 vars in 1137 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:20.565718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.577174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.639243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.655995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db93db9d-1e51-4e29-b4e3-c724472771ef None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.672340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44334075-02fb-4669-ba39-0e978e93f5ef None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.689144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.689755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.690024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.690289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.700553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.702001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.706010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.706594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.706865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.707165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.707783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:20.708078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.709416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:20.717559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:20.718490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:20.719400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:20.719400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:20.719400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:20.719400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 261/513] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:20.720035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.729509 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 253/514] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:20.729509 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.791835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.825292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5af26-c2f7-40ee-aca7-adea7ec04121 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.833927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7acd05-6230-4690-a603-5311512da210 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:20.857350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.857943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.858216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.858491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.859232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.860720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.861486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:20.867122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:20.867739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:20.868008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:20.868339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:20.876296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:20.877825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:20.888100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 262/515] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:19:20 2022] POST /identity/v3/users => generated 247 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:20.888100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:20.890675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:20.891631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:20.892521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 254/516] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:20.893043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.013374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6169f2-36a8-4021-810f-8afd5ae34803 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.018374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d016e0d-e482-42ce-9144-0d4eefc0a27b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.048298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.049663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.050465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.050755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.051114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.051759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:21.052053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.053408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.060928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.061653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.062028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.062395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.070750 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 255/517] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:21.070750 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.133104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.144814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.146210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.181708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dae74e2-f170-4cdf-a72d-0d73b29bbbd3 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.219984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.220610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.220883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.221151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.231166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.232777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:21.246225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:21.247421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 06 04:19:21.248540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:21.251948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 256/518] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/users/glance => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:21.252669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.257440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 263/519] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:21.258553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.289890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.360697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de619c8-db01-4d95-ad56-399389bb2f00 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.367259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295f9fd2-5221-4755-a626-884e74b45adc None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.375270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.404013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.404532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.404814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.405086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.405773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:21.406081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.406630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.407436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.407607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.408000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.408409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.416097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.417735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.426336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 257/520] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:21.427190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:21.430124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:21.431066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:21.432039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 264/521] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/users/neutron => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:21.432749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.525133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.539182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625b65a2-77c8-49f3-8910-a5221a85c507 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.548346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2aecda-b0ee-4fde-92f8-c4030c3d4282 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.577323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.577935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.578215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.578490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.585380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.585970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.586254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.586772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.587243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:21.587541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.588940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.590308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.591799 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.606399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 258/522] 10.222.0.33 () {56 vars in 1137 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:21.607237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:21.610239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:21.611587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:21.612892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:21.612892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:21.612892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:21.616943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 265/523] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:21.617782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.647419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.726776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27de69a-e56e-450d-92cc-2f6bf85bebc9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.729354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6c005b-386e-4f74-984f-0eb1d8196699 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.762577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.763192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.763467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.763737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.764444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=c8d2db883de44964b6e15b85a97726b5) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:21.764745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.766179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.766789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.767420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.767702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.767976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.769637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:21.769945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:21.771410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:21.773450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 259/524] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/role_assignments?user.id=c8d2db883de44964b6e15b85a97726b5&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:21.774243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.815231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.836527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 266/525] 10.222.0.33 () {56 vars in 1139 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:21.837270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:21.906073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:21.918554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120d6bd-e19d-4b5b-a46c-ee9b564daa4a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.952581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6f4e7-fcc0-455b-aabd-32b3b0071a58 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:21.969939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:21.971094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:21.971540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:21.972198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:21.999163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:22.000366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:22.000739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:22.001249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:22.002086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:22.002652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.004846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.023040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 267/526] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:22.023917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.065892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.067410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.099492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:22.147598 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb4ed5-85c4-4069-915c-3deaefe657e6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:22.180948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:22.187261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:22.188049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:22.188472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:22.188907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:22.189680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 260/527] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:22.190492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.270874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.272518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.323508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e63cdd-16f4-456d-b0d1-573a2c48d0e6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:22.333061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:22.374378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:22.375310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:22.375591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:22.375865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:22.381441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 268/528] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:22.382509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.384043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.385774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:22.408645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:22.409625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:22.410481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 261/529] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:22.410977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.490632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:22.519321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc705db-0ad0-4cf4-9730-78f9d32eddd1 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:22.541113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a0a76e-2087-4ae5-a264-7921f64fa80b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:22.567265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:22.568910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:22.569408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:22.570139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:22.579112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:22.579600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:22.579806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:22.580002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:22.588279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.591328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.591638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.593070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:22.608627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:22.609588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:22.610445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:22.610445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:22.610445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:22.610445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 262/530] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:22.611529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:22.612650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:22.613502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:22.613502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:22.613502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:22.615349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.615792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 269/531] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:22.617046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.633880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:22.744520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f353adb-6aed-4029-88c3-5ef64d5206d4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:22.746205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b2d117-3f7a-45a6-8b25-c0ca20ec53ad None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:22.754224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:22.787896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:22.788607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:22.788954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:22.789295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:22.789885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:22.790049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:22.790477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.790994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:22.791488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:22.791940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:22.792084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.793239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:22.793976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:22.796854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:22.822487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 263/532] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:22.822487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.830009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 270/533] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:22.831087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:22.914676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:22.950004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3389e7c3-a62f-466f-9830-1a8d1c5dca21 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:22.983728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ca1324-2c22-4831-adf0-274ed817f0da None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:23.012353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:23.013352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.013854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.014327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.017086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=61d1b78e81db4dbe987d788fd398e89d) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:23.017735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:23.019802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:23.027891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 271/534] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/role_assignments?user.id=61d1b78e81db4dbe987d788fd398e89d&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:23.029690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:23.051889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:23.051889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.051889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.051889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.052896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:23.053299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:23.054682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:23.101214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 264/535] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:23.107238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:23.183263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:23.240398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeeec91-0ab3-4ecb-a8fb-54e121b172be None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:23.285992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbb083b-1b9d-4003-a7df-7f4d8051dd27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:23.286618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb083b-1b9d-4003-a7df-7f4d8051dd27 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.286892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb083b-1b9d-4003-a7df-7f4d8051dd27 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.287361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb083b-1b9d-4003-a7df-7f4d8051dd27 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.289268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:23.299333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbb083b-1b9d-4003-a7df-7f4d8051dd27 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:23.299333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbb083b-1b9d-4003-a7df-7f4d8051dd27 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:23.309556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d171f-4138-40c1-bb76-454d31e57178 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-9cbb083b-1b9d-4003-a7df-7f4d8051dd27 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:23.327914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:23.328828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:23.329680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:23.332661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 272/536] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:23.334132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c927ef0f-f658-496a-9a8a-49bae703d12a None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.335316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c927ef0f-f658-496a-9a8a-49bae703d12a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.335858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c927ef0f-f658-496a-9a8a-49bae703d12a None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.345022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 273/537] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:23 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:23.349805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c927ef0f-f658-496a-9a8a-49bae703d12a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:23.372355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea8cba7-ee92-4b28-9442-b9649bacb7b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:23.373572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea8cba7-ee92-4b28-9442-b9649bacb7b1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.374065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea8cba7-ee92-4b28-9442-b9649bacb7b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.375022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea8cba7-ee92-4b28-9442-b9649bacb7b1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.388309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea8cba7-ee92-4b28-9442-b9649bacb7b1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:23.393222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea8cba7-ee92-4b28-9442-b9649bacb7b1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-0ea8cba7-ee92-4b28-9442-b9649bacb7b1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:23.420710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:23.421650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:23.422504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:23.429670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 265/538] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:23.431340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599096c9-6d3a-43f1-9432-dd5a845fca1b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.431826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599096c9-6d3a-43f1-9432-dd5a845fca1b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.432292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599096c9-6d3a-43f1-9432-dd5a845fca1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.469718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-599096c9-6d3a-43f1-9432-dd5a845fca1b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:23.472645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-599096c9-6d3a-43f1-9432-dd5a845fca1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:23.550616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c927ef0f-f658-496a-9a8a-49bae703d12a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:23.584804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 266/539] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:23.584804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599096c9-6d3a-43f1-9432-dd5a845fca1b None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:23.608208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:23.610445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:23.611348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.614458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.615326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.616109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:23.616602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:23.618472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:23.640596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 274/540] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:23.646391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:23.746001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:23.764778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599096c9-6d3a-43f1-9432-dd5a845fca1b None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:23.811763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc608d1-9ac8-4da7-8251-cdc1cd07ad5a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:23.835869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:23.836423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.836696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.836964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.837669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:23.841030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:23.842697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:23.867873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 267/541] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:23.874687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:23.883678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:23.884295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:23.884571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:23.884839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:23.885565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=87829678318948f3b81bd107076bdfa7) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:23.885868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:23.888400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:23.896541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 275/542] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:23 2022] GET /identity/v3/role_assignments?user.id=87829678318948f3b81bd107076bdfa7&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 259 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:23.903406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:24.028984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:24.083144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424fe5de-0559-4c5f-8838-2b405a977594 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:24.110847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a5d361-0d0b-4825-8bc7-c36418f82c74 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:24.149306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:24.154529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97700922-ab00-43a3-bb90-8ee042affc3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:24.155340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97700922-ab00-43a3-bb90-8ee042affc3e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.155589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97700922-ab00-43a3-bb90-8ee042affc3e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.155854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97700922-ab00-43a3-bb90-8ee042affc3e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.156581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97700922-ab00-43a3-bb90-8ee042affc3e None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=b34c61ebdb9147d5b3d66cbfa556f38e) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:24.156880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97700922-ab00-43a3-bb90-8ee042affc3e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:24.158295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97700922-ab00-43a3-bb90-8ee042affc3e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:24.164114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 268/543] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:23 2022] GET /identity/v3/role_assignments?user.id=b34c61ebdb9147d5b3d66cbfa556f38e&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 259 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:24.165094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6ba19-476f-471a-983e-cf39b8b95dcc None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.165370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6ba19-476f-471a-983e-cf39b8b95dcc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.165639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6ba19-476f-471a-983e-cf39b8b95dcc None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.171534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 269/544] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:24.171797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:24.172417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.172700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.172855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384000d5-3589-49aa-88e6-e44010a488f4 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.172965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.173137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384000d5-3589-49aa-88e6-e44010a488f4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.173468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384000d5-3589-49aa-88e6-e44010a488f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.185275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:24.190910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:24.191913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:24.192722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 276/545] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:24.198350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-384000d5-3589-49aa-88e6-e44010a488f4 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:24.200318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-384000d5-3589-49aa-88e6-e44010a488f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:24.204129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:24.276560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 270/546] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:24.427351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721ae910-e86e-457f-a9ab-b6cd34161b8c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:24.468161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:24.468765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.469043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.469317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.469985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:24.470284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:24.471697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:24.485469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 277/547] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:24.490942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384000d5-3589-49aa-88e6-e44010a488f4 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:24.591017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:24.600299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:24.635899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384000d5-3589-49aa-88e6-e44010a488f4 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:24.690927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:24.694853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:24.695641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.695954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.696230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.716845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:24.718295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:24.747016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:24.747983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:24.748841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:24.748841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:24.748841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:24.749965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 271/548] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:24.754307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:24.803996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c4c7e3-8c92-4920-80fe-86e534aff14e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:24.842148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:24.843613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.844170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.844744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.856340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:24.862350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:24.863762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:24.864738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:24.867382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 278/549] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:24.875093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:24.885438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5276185-6ed6-45c1-a674-3bef3ece5140 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:24.941842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:24.974077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:24.974953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:24.975396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:24.975811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:24.976737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:24.977286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:24.979402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:24.994267 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 272/550] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:25.001598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:25.107143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211e1bd-5732-4692-9cad-f139c26f9281 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.118430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aee6eb-a122-4c4b-801d-56f5c3a47cb5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.141851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:25.142462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.142739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.143038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.143706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:25.144007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:25.145764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:25.154488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:25.154786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 279/551] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:25.161395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:25.162270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.163182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.163277 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:25.163367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.172630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:25.174397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:25.189876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:25.190981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:25.191974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:25.194463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 273/552] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:25 2022] GET /identity/v3/users/swift => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:25.200766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:25.254478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:25.323142 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca3f49-1809-4da2-ba77-7aa5f9c007f8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.360614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:25.361582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.362301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.362794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.363736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:25.364238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:25.366177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:25.387817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 274/553] 10.222.0.33 () {56 vars in 1135 bytes} [Thu Oct 6 04:19:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:25.393730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:25.461185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5f6f73-56e9-4349-ad85-106c0c673f4b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.498098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:25.501985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc13a0e-3bf6-45b7-8cca-8009b6053037 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.539550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:25.540520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.541015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.541498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.545663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:25.546452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.547050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.547494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.558795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:25.561306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:25.584027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:25.585022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:25.585900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:25.585900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:25.585900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:25.586798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 280/554] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 424 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:25.591106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:25.627326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:25.659563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:25.661017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:25.794481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe666aa-a120-4da3-b39e-cc17e056def5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.812882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:25.813193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 275/555] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 06 04:19:25.820196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:25.848166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ba1a07-770b-4691-b056-2d4cf7b1d0cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:25.848771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba1a07-770b-4691-b056-2d4cf7b1d0cf None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.849059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba1a07-770b-4691-b056-2d4cf7b1d0cf None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.849329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba1a07-770b-4691-b056-2d4cf7b1d0cf None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.849966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba1a07-770b-4691-b056-2d4cf7b1d0cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:25.850268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba1a07-770b-4691-b056-2d4cf7b1d0cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:25.851739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba1a07-770b-4691-b056-2d4cf7b1d0cf None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:25.865102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 281/556] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:25.866396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310db2a-5563-43eb-b147-314e5d8a32c2 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.866676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310db2a-5563-43eb-b147-314e5d8a32c2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.866945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310db2a-5563-43eb-b147-314e5d8a32c2 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.873353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 282/557] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:25.874260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9310db2a-5563-43eb-b147-314e5d8a32c2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:25.939598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0536f260-95ef-4c81-b263-520c5d3e8664 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.959109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:25.983075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f52d2c-cb25-4924-99af-ccb28607b7f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:25.983725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f52d2c-cb25-4924-99af-ccb28607b7f6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:25.984589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f52d2c-cb25-4924-99af-ccb28607b7f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:25.984864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f52d2c-cb25-4924-99af-ccb28607b7f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:25.991613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9310db2a-5563-43eb-b147-314e5d8a32c2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:25.998569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f52d2c-cb25-4924-99af-ccb28607b7f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.000046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f52d2c-cb25-4924-99af-ccb28607b7f6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-12f52d2c-cb25-4924-99af-ccb28607b7f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:26.017990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:26.023559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:26.024746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:26.024746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:26.024746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:26.024746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 276/558] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:26.024746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3f74e7-4957-4151-91e3-93cc302c794c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.024746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3f74e7-4957-4151-91e3-93cc302c794c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.024746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3f74e7-4957-4151-91e3-93cc302c794c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.039428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.040403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.040825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.041273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.044426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-5e3f74e7-4957-4151-91e3-93cc302c794c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:26.046370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5e3f74e7-4957-4151-91e3-93cc302c794c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:26.050306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.051934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:26.066370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:26.067503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:26.068438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:26.070943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 283/559] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:19:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:26.071742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.107786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:26.143420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 277/560] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:26.143994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3f74e7-4957-4151-91e3-93cc302c794c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.250647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa91655-cf49-41ae-861f-6e2833f2eac0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:26.317671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3f74e7-4957-4151-91e3-93cc302c794c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:26.320900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.321527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.321802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.322063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.322719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:26.323039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.324683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.326594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:26.356418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 284/561] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:26.363090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.392909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.393520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.393799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.394073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.394806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:26.395136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.396563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.438874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 278/562] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:26.447992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.496286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:26.510586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4614082-fc82-48cd-962b-d143f7c8522e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:26.550722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.551383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.551661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.551929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.554734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f685d-2bff-4620-a136-35c7e6da5df1 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:26.559244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.560695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:26.575379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:26.576448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:26.577454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 285/563] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:26.578085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.593112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16659476-f6f9-4dfa-9209-3a82fc6c7e08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.593732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16659476-f6f9-4dfa-9209-3a82fc6c7e08 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.594004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16659476-f6f9-4dfa-9209-3a82fc6c7e08 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.594268 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16659476-f6f9-4dfa-9209-3a82fc6c7e08 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.604648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16659476-f6f9-4dfa-9209-3a82fc6c7e08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.606103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16659476-f6f9-4dfa-9209-3a82fc6c7e08 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-16659476-f6f9-4dfa-9209-3a82fc6c7e08 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:26.621733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:26.622904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:26.623843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:26.623843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:26.623843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:26.623843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 279/564] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:26.624285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba724965-14e9-4158-a571-83e1d02b98c8 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.624562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba724965-14e9-4158-a571-83e1d02b98c8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.624825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba724965-14e9-4158-a571-83e1d02b98c8 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.626285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 280/565] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:26.626986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba724965-14e9-4158-a571-83e1d02b98c8 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.648021 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:26.700906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021e4e0b-38de-4e14-a062-eb61c50cd1c3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:26.733081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:26.738036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.738644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.738919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.739215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.744064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba724965-14e9-4158-a571-83e1d02b98c8 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:26.750243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:26.755442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:26.756384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:26.757236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 286/566] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:26.757930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.787772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236e8eed-c59c-48aa-a595-0ec610fc4bf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.788378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236e8eed-c59c-48aa-a595-0ec610fc4bf2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.788686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236e8eed-c59c-48aa-a595-0ec610fc4bf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.789172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236e8eed-c59c-48aa-a595-0ec610fc4bf2 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.789881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236e8eed-c59c-48aa-a595-0ec610fc4bf2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:26.790174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236e8eed-c59c-48aa-a595-0ec610fc4bf2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.791639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236e8eed-c59c-48aa-a595-0ec610fc4bf2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.806918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 281/567] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:26.808674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b5bfc0-2bfc-4237-99de-0c06ec4724fb None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.808953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b5bfc0-2bfc-4237-99de-0c06ec4724fb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.809222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b5bfc0-2bfc-4237-99de-0c06ec4724fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.827278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-83b5bfc0-2bfc-4237-99de-0c06ec4724fb None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:26.828379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-83b5bfc0-2bfc-4237-99de-0c06ec4724fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:26.865338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dac82d3-b9e4-484f-955f-333a2b516fda None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:26.883769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:26.904540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc57c76-d7db-4bb8-9067-ee56354e76ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:26.904729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 282/568] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:26.905141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc57c76-d7db-4bb8-9067-ee56354e76ec None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.905415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc57c76-d7db-4bb8-9067-ee56354e76ec None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.905513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b5bfc0-2bfc-4237-99de-0c06ec4724fb None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.905683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc57c76-d7db-4bb8-9067-ee56354e76ec None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.906307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc57c76-d7db-4bb8-9067-ee56354e76ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:26.906604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc57c76-d7db-4bb8-9067-ee56354e76ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:26.908000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc57c76-d7db-4bb8-9067-ee56354e76ec None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:26.924171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 287/569] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:26.924171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020090f-cf38-49ce-9d82-34252dfef77a None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:26.924171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020090f-cf38-49ce-9d82-34252dfef77a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:26.924171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020090f-cf38-49ce-9d82-34252dfef77a None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:26.924171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 288/570] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:26.924171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0020090f-cf38-49ce-9d82-34252dfef77a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:26.999855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.019398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b5bfc0-2bfc-4237-99de-0c06ec4724fb None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.031787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0020090f-cf38-49ce-9d82-34252dfef77a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.057180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.057777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.058564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.058911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.059507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:27.059797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.061128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.073330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 283/571] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:27.073545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8e8d40-b7b1-47fc-97ce-fe62806012a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.074184 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.074265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8e8d40-b7b1-47fc-97ce-fe62806012a6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.074265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8e8d40-b7b1-47fc-97ce-fe62806012a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.074265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8e8d40-b7b1-47fc-97ce-fe62806012a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.074265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8e8d40-b7b1-47fc-97ce-fe62806012a6 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=2e280c4b418e4aafb822a617424c7e11) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:27.074265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8e8d40-b7b1-47fc-97ce-fe62806012a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.074265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8e8d40-b7b1-47fc-97ce-fe62806012a6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.086911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 289/572] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:26 2022] GET /identity/v3/role_assignments?user.id=2e280c4b418e4aafb822a617424c7e11&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:27.086911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c27bd96-7c93-4b4b-bd66-377c3fa3402d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.086911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c27bd96-7c93-4b4b-bd66-377c3fa3402d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.086911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c27bd96-7c93-4b4b-bd66-377c3fa3402d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.110631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9c27bd96-7c93-4b4b-bd66-377c3fa3402d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:27.112240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9c27bd96-7c93-4b4b-bd66-377c3fa3402d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:27.156982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.199411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71a4f3-215e-4290-b12e-9a52cd607037 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.219743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 290/573] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:27.220840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c27bd96-7c93-4b4b-bd66-377c3fa3402d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.249983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.250659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.250982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.251395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.259272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:27.273791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:27.275087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:27.276240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:27.279429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 284/574] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:27.280266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.287337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.352019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c27bd96-7c93-4b4b-bd66-377c3fa3402d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.396073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43377a58-44dd-4e20-9511-e939a7700769 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.401095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.402186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.402662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.403109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.418165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.420494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.432003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.432709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.432961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.433236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:27.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:27.442676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:27.443836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:27.443836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:27.443836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:27.444915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:27.445702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 291/575] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:27.446674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:27.452561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:27.453747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:27.454825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:27.456530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 285/576] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:27.461331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.488049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.531082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.571886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c7ec5-daa7-49eb-a0d5-4226cbb6912b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.587845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a639c7-319b-4a44-a30c-ab9c9fedf401 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.609465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.609982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.610252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.610515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.611169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:27.611465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.613060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.629505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 286/577] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:27.629505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.638678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.639474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.639756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.640030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.640731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:27.641037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.642469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.655730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 292/578] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:27.656839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.736001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.738539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e2687-7cbb-44bc-be33-dcfe3cbd5a36 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.756773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4ec01f-9e69-4162-a2f9-ff01c7925982 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.764642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.777770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.778369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.778649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.778919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.779606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:27.779904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.781246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.789725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 287/579] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:27.790733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.797062 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.797899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.798314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.798747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.811028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:27.817000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:27.818124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:27.819182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:27.820167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 293/580] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:27.820167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.912448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861c85d-77f4-4a47-b0f4-1a1118065738 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.927112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9823dafd-4671-44f2-988e-c48d20a70577 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:27.955462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.956094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.956383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.956967 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.966130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:27.966865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:27.967012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.967188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:27.967452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/c8d2db883de44964b6e15b85a97726b5/roles/bcc8e52e7c184e5984637371b5614639` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:27.968427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:27.978884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:27.979500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:27.980544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 288/581] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/users/cinder => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:27.981510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:27.988436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=c8d2db883de44964b6e15b85a97726b5, role_id=bcc8e52e7c184e5984637371b5614639)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:27.990330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.003278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 294/582] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:27 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/c8d2db883de44964b6e15b85a97726b5/roles/bcc8e52e7c184e5984637371b5614639 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:28.004458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.047580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:28.121732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e9eb30-ea33-4c42-9725-420023984cd5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.146980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b13a90-24f4-4cea-a1b1-f96a67609da1 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.172235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.173534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.174009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.174669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.187459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.188074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.188356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.188629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.189311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:28.189718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.190664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.191231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.192623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.203004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 295/583] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:28.204039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:28.213963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:28.214953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:28.215890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:28.215890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:28.215890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:28.220847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 289/584] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:28.222201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.282003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:28.342499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:28.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf22cf3c-15c5-485e-9494-6d343e28e2a5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.358644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134f7f35-ecf2-45fe-a12f-aa3b27983377 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.397362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.397988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.398230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.398498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.399241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:28.399545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.400983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.403041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.404977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.405488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.405971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.420156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 296/585] 10.222.0.33 () {56 vars in 1137 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:28.420808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.421060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.422881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:28.443139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:28.444238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:28.445246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:28.445246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:28.445246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:28.447754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 290/586] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:28.449036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.533538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fa7c8a-fe40-45d6-9d64-05e8b3cc9a31 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.546556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:28.578301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.579453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.580286 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.580843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.581754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:28.582453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.584713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.586110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38a9988-e130-4041-946e-f0d5f6a887be None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.600713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 297/587] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:28.602523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.624893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.625713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.626084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.626489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.638218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.639779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:28.655363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:28.656550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:28.657672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:28.657672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:28.657672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:28.659091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 291/588] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:28.660023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.662950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:28.733030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e6891d-00f3-4ef3-bfa6-10c2cfd7e945 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.778794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.779960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.780425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.781105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.782320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:28.783077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.784630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.784828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44393d9a-6298-48b3-8ef4-50fbafebdaf8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.804388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 298/589] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:28.805547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.829885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.830842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.831342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.831778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.844557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:28.849679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.853028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:28.867469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:28.868788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 06 04:19:28.870004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:28.872798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 292/590] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/users/neutron => generated 85 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:28.873917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:28.924797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d5b6a-067b-4bcb-827b-44a2eccb30cd None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:28.954914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:28.962198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:28.963084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:28.963456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:28.963824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:28.964680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:28.965071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:28.966541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:28.979056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 299/591] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:28.980415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.007127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637759a-3d9e-4369-94ca-9a006f6330d7 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.051150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.052293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.052730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.053148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.063225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.064866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:29.080126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:29.081321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 06 04:19:29.082503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:29.084818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:29.086056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 293/592] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/users/glance => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:29.087179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.097020 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca97be2-15dc-499a-bd1e-4633969f7ad5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.132960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.133672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.134093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.134971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.135309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:29.135712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.137405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.156539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 300/593] 10.222.0.33 () {56 vars in 1139 bytes} [Thu Oct 6 04:19:28 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:29.157585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.216792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd0524c-8583-4c18-88da-27a1e7b522c6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.252132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:29.267621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.268494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.268929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.269102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf5423f-1f93-4d09-8478-4e3d4fdab463 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.269498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.277564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.279525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:29.292797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:29.294007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:29.294986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:29.297752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 294/594] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:29.299005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.309500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.310317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.310732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.311159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.312123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:29.312508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.314444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.334189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 301/595] 10.222.0.33 () {56 vars in 1137 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:29.335319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.345335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:29.419395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feba886-4b5b-422b-9f5b-4453a00f3400 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.443121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1ac5ec-4dc9-414c-b534-73e0d76fcc84 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.454451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79364f92-09df-4fb3-a14a-8a1cc8033e7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.455205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79364f92-09df-4fb3-a14a-8a1cc8033e7d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.455514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79364f92-09df-4fb3-a14a-8a1cc8033e7d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.455742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79364f92-09df-4fb3-a14a-8a1cc8033e7d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.467893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79364f92-09df-4fb3-a14a-8a1cc8033e7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.469316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79364f92-09df-4fb3-a14a-8a1cc8033e7d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.478028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.478634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.478923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.479217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.480063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:29.480378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.481857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-79364f92-09df-4fb3-a14a-8a1cc8033e7d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:29.485519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:29.486655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:29.488055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:29.488055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:29.488055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:29.494281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 302/596] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:29.494520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.494908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 295/597] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:29.496233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9bcbf4-c516-48f2-a1ec-6784b4af7f2d None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.496516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9bcbf4-c516-48f2-a1ec-6784b4af7f2d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.496792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9bcbf4-c516-48f2-a1ec-6784b4af7f2d None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.498186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 296/598] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:29.499115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9bcbf4-c516-48f2-a1ec-6784b4af7f2d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.537374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:29.572791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:29.610358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dc9a2f-ca0a-4105-89f9-07252b78990b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.610575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9bcbf4-c516-48f2-a1ec-6784b4af7f2d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.645832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef80008-7b5d-4221-9c37-d736d0b66b7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.646469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef80008-7b5d-4221-9c37-d736d0b66b7e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.646742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef80008-7b5d-4221-9c37-d736d0b66b7e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.647040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef80008-7b5d-4221-9c37-d736d0b66b7e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.647958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.648554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.648830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.649094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.649728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:29.650025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.651424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.658410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef80008-7b5d-4221-9c37-d736d0b66b7e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.659887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef80008-7b5d-4221-9c37-d736d0b66b7e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.668727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 297/599] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:29.668727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-8ef80008-7b5d-4221-9c37-d736d0b66b7e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:29.674771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:29.675739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:29.681073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:29.681073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:29.681073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:29.681073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 303/600] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:29.681073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80fdf23-f28e-48bf-b42b-21f0b34380c9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.681073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80fdf23-f28e-48bf-b42b-21f0b34380c9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.681073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80fdf23-f28e-48bf-b42b-21f0b34380c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.697634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f80fdf23-f28e-48bf-b42b-21f0b34380c9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:29.698939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f80fdf23-f28e-48bf-b42b-21f0b34380c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:29.769054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 304/601] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:29.770900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:29.778291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eff64b-8de0-416f-b1c7-65e774e9e7c2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.779860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80fdf23-f28e-48bf-b42b-21f0b34380c9 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.815987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.816878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.817532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.817775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/61d1b78e81db4dbe987d788fd398e89d/roles/bcc8e52e7c184e5984637371b5614639` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.836205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=61d1b78e81db4dbe987d788fd398e89d, role_id=bcc8e52e7c184e5984637371b5614639)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.837639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.849161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:29.853485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 298/602] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:29 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/61d1b78e81db4dbe987d788fd398e89d/roles/bcc8e52e7c184e5984637371b5614639 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:29.854444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:29.915601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80fdf23-f28e-48bf-b42b-21f0b34380c9 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.960081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:29.961213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:29.961736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:29.962501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:29.972624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:29.974503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:29.974904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31fc149-64e5-408b-b9d3-902d0c1a2f4d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:29.988123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:29.989071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:29.989995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:29.992571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 305/603] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:29.999176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.014173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.014753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.015057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.015328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.015978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:30.016423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.018025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.037417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 299/604] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:30.039214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.044783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:30.122676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc7ad2d-be87-43a7-a43d-e57b85a38e49 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.133461 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:30.159856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c36513-a3b6-435e-b64f-e3b3eb2f4f54 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.166890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.168636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.168921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.169191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.169887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:30.170628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.172535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.189195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 306/605] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:30.190043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.199859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.200585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.200932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.201277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.208233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.214869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:30.230665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:30.231654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:30.232562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:30.235036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 300/606] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:30.236913 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.305247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4e1f50-6803-40cd-9af2-9361ecfa827c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.320210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:30.346285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.346942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.347245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.347515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.348376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703cbf7b-e218-4534-8102-f34059fec60f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.360567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:30.366589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:30.367596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:30.368449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:30.372894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 307/607] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:30.373602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.391478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.392558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.393007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.393460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/87829678318948f3b81bd107076bdfa7/roles/bcc8e52e7c184e5984637371b5614639` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.404731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:30.410295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=87829678318948f3b81bd107076bdfa7, role_id=bcc8e52e7c184e5984637371b5614639)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.412758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.430149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 301/608] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:30 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/87829678318948f3b81bd107076bdfa7/roles/bcc8e52e7c184e5984637371b5614639 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:30.431416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.498642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d2904f-9965-4f59-af32-bedaf59f9baa None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.547815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fe5a0e-ef83-43e1-a05a-da7f2bda0ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.548714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fe5a0e-ef83-43e1-a05a-da7f2bda0ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.548997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fe5a0e-ef83-43e1-a05a-da7f2bda0ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.549271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fe5a0e-ef83-43e1-a05a-da7f2bda0ae4 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.550293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fe5a0e-ef83-43e1-a05a-da7f2bda0ae4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:30.550602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fe5a0e-ef83-43e1-a05a-da7f2bda0ae4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.552975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fe5a0e-ef83-43e1-a05a-da7f2bda0ae4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.566424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b77b6b-e49c-4cd4-aef6-72b11b5dad0e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.577019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 308/609] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:30.579123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7734c1f2-a529-44b1-abd4-5fc0db199667 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.579534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7734c1f2-a529-44b1-abd4-5fc0db199667 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.579910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7734c1f2-a529-44b1-abd4-5fc0db199667 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.583149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 309/610] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:30.584223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7734c1f2-a529-44b1-abd4-5fc0db199667 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.588556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:30.618627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d81a58-835b-4baa-99e1-2a03a8a94f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.619261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d81a58-835b-4baa-99e1-2a03a8a94f6a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.619534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d81a58-835b-4baa-99e1-2a03a8a94f6a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.619799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d81a58-835b-4baa-99e1-2a03a8a94f6a None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.620980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d81a58-835b-4baa-99e1-2a03a8a94f6a None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:30.623272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d81a58-835b-4baa-99e1-2a03a8a94f6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.624779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d81a58-835b-4baa-99e1-2a03a8a94f6a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.637972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 302/611] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:30.640187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b1fd62-4827-44f3-a056-e2bb964816e5 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.640662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b1fd62-4827-44f3-a056-e2bb964816e5 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.641111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b1fd62-4827-44f3-a056-e2bb964816e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.662009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-55b1fd62-4827-44f3-a056-e2bb964816e5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:30.663556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-55b1fd62-4827-44f3-a056-e2bb964816e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:30.717387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:30.731161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7734c1f2-a529-44b1-abd4-5fc0db199667 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.755355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 303/612] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:30.756441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b1fd62-4827-44f3-a056-e2bb964816e5 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:30.784905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.785673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.791389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.792503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/b34c61ebdb9147d5b3d66cbfa556f38e/roles/bcc8e52e7c184e5984637371b5614639` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.819033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=b34c61ebdb9147d5b3d66cbfa556f38e, role_id=bcc8e52e7c184e5984637371b5614639)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.820487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.859169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 310/613] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:30 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/b34c61ebdb9147d5b3d66cbfa556f38e/roles/bcc8e52e7c184e5984637371b5614639 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:19:30.863793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b1fd62-4827-44f3-a056-e2bb964816e5 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:30.873867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:30.903882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:30.904858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:30.905323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:30.905829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:30.919148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:30.921071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:30.939055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:30.940085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:30.941020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:30.941020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:30.941020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:30.942982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 304/614] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:30.952070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:31.097525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d547f9ff-6649-4c40-b5de-14725f6fdeba None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:31.136572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:31.137206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:31.137481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:31.137790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:31.138392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:31.138694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:31.140109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:31.147421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:31.154331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 311/615] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:31.159375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:31.245089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:31.322807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18652-8513-4e5e-9cc7-34edf527d2c6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:31.357813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92451943-9bae-4c96-995f-390f39bf3186 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:31.406384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:31.407470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:31.408588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:31.409537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:31.421187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:31.422037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:31.422470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:31.422885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:31.426494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:31.430105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:31.444617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:31.450230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:31.451370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:31.452444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:31.457039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:31.458175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:31.459379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:31.466237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 312/616] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:31.466672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 305/617] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:31 2022] GET /identity/v3/users/swift => generated 83 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:31.468836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:31.474218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:31.496376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:31.584271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1debfe-56eb-457d-b7c0-73e36ae0a572 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:31.611324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:31.619976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:31.621061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:31.621541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:31.622051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:31.623012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:31.623461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:31.624948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:31.655389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92881232-7427-4bb0-9647-a1c0a3e57658 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:31.661730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 306/618] 10.222.0.33 () {56 vars in 1135 bytes} [Thu Oct 6 04:19:31 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:31.662988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:31.691082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:31.692044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:31.692517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:31.692982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:31.693863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:31.694356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:31.696162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:31.705789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 313/619] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:31.717904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:31.814732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:31.860339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde3961f-dfb4-4bdf-9414-9fb075714be8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:31.899771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2804e1-d2e2-4e4d-b7fc-09e478302908 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:31.928840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:31.929363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:31.929633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:31.929897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:31.943482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:31.953608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:31.955089 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:31.972615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:31.973627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:31.974101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:31.974565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:31.976290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:31.977275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:31.978188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:31.978188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:31.978188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:31.978188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 307/620] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:31.984315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.004046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:32.005851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:32.029081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:32.030078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:32.030995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:32.030995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:32.030995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:32.038409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 314/621] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:32.044445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.187772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c545b6de-28d3-44db-866a-10c1e211bad7 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:32.218466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c4fb03-f4de-4c5a-93cb-db60b74064b3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:32.248080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:32.264802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:32.265419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.265700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.265971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.266625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:32.267128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:32.268530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:32.297786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:32.298399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.298673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.298942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.299608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:32.299911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:32.301294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:32.303088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 308/622] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:32.310031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.325582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 315/623] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:32.332310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.368836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:32.427875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fb28d2-c15b-4b3e-8c57-64a8f06befc4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:32.441163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32111a15-6fcf-4cd6-9d59-b88f1bd10157 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:32.467481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7750545e-90fd-4dc6-aa35-b238cb9379aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:32.468570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7750545e-90fd-4dc6-aa35-b238cb9379aa None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.469158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7750545e-90fd-4dc6-aa35-b238cb9379aa None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.469633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7750545e-90fd-4dc6-aa35-b238cb9379aa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.476604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d4fe80-fb40-4911-affb-04054f7f1926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:32.477219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4fe80-fb40-4911-affb-04054f7f1926 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.477490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4fe80-fb40-4911-affb-04054f7f1926 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.477755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4fe80-fb40-4911-affb-04054f7f1926 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.478233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7750545e-90fd-4dc6-aa35-b238cb9379aa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:32.480519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7750545e-90fd-4dc6-aa35-b238cb9379aa None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:32.485788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d4fe80-fb40-4911-affb-04054f7f1926 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:32.487514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d4fe80-fb40-4911-affb-04054f7f1926 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7750545e-90fd-4dc6-aa35-b238cb9379aa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:32.493492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:32.494399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:32.495405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:32.497801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 309/624] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:32.499204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91842e9f-baa2-4827-ab6a-1d5b0df40e80 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.499705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91842e9f-baa2-4827-ab6a-1d5b0df40e80 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.500170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91842e9f-baa2-4827-ab6a-1d5b0df40e80 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.501594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 310/625] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-56d4fe80-fb40-4911-affb-04054f7f1926 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:32.503446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:32.504366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:32.505330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:32.508163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 316/626] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:19:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:32.508454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91842e9f-baa2-4827-ab6a-1d5b0df40e80 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.510212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74583fb8-3daf-447f-9298-f21f6b74fa90 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.513095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74583fb8-3daf-447f-9298-f21f6b74fa90 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.513594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74583fb8-3daf-447f-9298-f21f6b74fa90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.538243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-74583fb8-3daf-447f-9298-f21f6b74fa90 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:32.540140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-74583fb8-3daf-447f-9298-f21f6b74fa90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:32.572433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:32.639012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 317/627] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:32.639012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74583fb8-3daf-447f-9298-f21f6b74fa90 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.654747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91842e9f-baa2-4827-ab6a-1d5b0df40e80 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:32.678849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:32.720286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:32.720807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.721079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.721345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.722031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:32.722332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:32.723776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:32.748887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 311/628] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:32.756686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.795884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74583fb8-3daf-447f-9298-f21f6b74fa90 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:32.833957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:32.835015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:32.835489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:32.835953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:32.836919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:32.837416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:32.840122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:32.861456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 318/629] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:19:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:32.867835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:32.949113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cff634-aad2-4c99-b0c8-e7d144225ac4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:32.968010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.005872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.007430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c1774f-3d74-4db8-9dd9-f91e36731cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.008120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c1774f-3d74-4db8-9dd9-f91e36731cc9 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.008399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c1774f-3d74-4db8-9dd9-f91e36731cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.008668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c1774f-3d74-4db8-9dd9-f91e36731cc9 None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/2e280c4b418e4aafb822a617424c7e11/roles/bcc8e52e7c184e5984637371b5614639` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.027926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c1774f-3d74-4db8-9dd9-f91e36731cc9 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=2e280c4b418e4aafb822a617424c7e11, role_id=bcc8e52e7c184e5984637371b5614639)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:33.029217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bcefe3-4d2b-4678-9ee4-43acc77b61af None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.029328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c1774f-3d74-4db8-9dd9-f91e36731cc9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:33.050994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 312/630] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:32 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/2e280c4b418e4aafb822a617424c7e11/roles/bcc8e52e7c184e5984637371b5614639 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:33.050994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c3918b-b0c2-4230-bb49-81d179f43f43 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.050994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c3918b-b0c2-4230-bb49-81d179f43f43 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.050994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c3918b-b0c2-4230-bb49-81d179f43f43 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.050994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 313/631] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:33.050994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c3918b-b0c2-4230-bb49-81d179f43f43 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.076945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.077746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.078027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.078307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.177327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c3918b-b0c2-4230-bb49-81d179f43f43 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.179241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:33.181737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:33.223049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52caafa-7e90-41fc-88c5-69222c3763cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.224039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52caafa-7e90-41fc-88c5-69222c3763cc None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.224469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52caafa-7e90-41fc-88c5-69222c3763cc None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.224735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52caafa-7e90-41fc-88c5-69222c3763cc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.240685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52caafa-7e90-41fc-88c5-69222c3763cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c52caafa-7e90-41fc-88c5-69222c3763cc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:33.248826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:33.249788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:33.250653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 314/632] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:33.252084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eeb37c-975e-4f98-b018-203ecf60e276 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.252360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eeb37c-975e-4f98-b018-203ecf60e276 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.252633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46eeb37c-975e-4f98-b018-203ecf60e276 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.274486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-46eeb37c-975e-4f98-b018-203ecf60e276 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:33.275623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-46eeb37c-975e-4f98-b018-203ecf60e276 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:33.291392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.293048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.319083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 319/633] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 06 04:19:33.320164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.365692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 315/634] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:33.366864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eeb37c-975e-4f98-b018-203ecf60e276 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.434104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035e211-ede8-4914-ab04-eaeecc19bb41 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.472621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.473338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.473685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.474026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.474851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:33.475249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:33.476690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:33.487009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 320/635] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:33.489040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46eeb37c-975e-4f98-b018-203ecf60e276 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.500217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.525712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35045b7-e87f-4102-950b-9ddb216a1f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.527363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35045b7-e87f-4102-950b-9ddb216a1f56 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.527841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35045b7-e87f-4102-950b-9ddb216a1f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.528338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35045b7-e87f-4102-950b-9ddb216a1f56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.540853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35045b7-e87f-4102-950b-9ddb216a1f56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:33.542930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35045b7-e87f-4102-950b-9ddb216a1f56 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:33.552030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-b35045b7-e87f-4102-950b-9ddb216a1f56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:33.561603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:33.562583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:33.563564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:33.563564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:33.563564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:33.563564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 316/636] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:33.568767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bac7478-3cf8-4748-9ce8-3991c899382d None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.568767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bac7478-3cf8-4748-9ce8-3991c899382d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.568767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bac7478-3cf8-4748-9ce8-3991c899382d None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.569026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 317/637] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:33.569583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bac7478-3cf8-4748-9ce8-3991c899382d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.588278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.652832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2da5834-6d22-4451-8b84-ff4b36ca73f2 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.683342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bac7478-3cf8-4748-9ce8-3991c899382d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.689949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c758d22-44b8-4356-bd6b-6d0dc6a8ea94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.690588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c758d22-44b8-4356-bd6b-6d0dc6a8ea94 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.690865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c758d22-44b8-4356-bd6b-6d0dc6a8ea94 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.691168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c758d22-44b8-4356-bd6b-6d0dc6a8ea94 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.703561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c758d22-44b8-4356-bd6b-6d0dc6a8ea94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:33.705090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c758d22-44b8-4356-bd6b-6d0dc6a8ea94 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-3c758d22-44b8-4356-bd6b-6d0dc6a8ea94 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:33.720080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:33.721102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:33.721998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:33.721998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:33.721998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:33.721998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 321/638] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:33.722710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e6c59-f6d6-40e6-b7ed-ebc4275b723b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.722710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e6c59-f6d6-40e6-b7ed-ebc4275b723b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.722941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e6c59-f6d6-40e6-b7ed-ebc4275b723b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.723062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.723682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.723958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.724225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.724863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:33.725161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:33.726531 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:33.740307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 318/639] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:33.741137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.742003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9b2e6c59-f6d6-40e6-b7ed-ebc4275b723b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:33.743291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9b2e6c59-f6d6-40e6-b7ed-ebc4275b723b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:33.807115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.815010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 322/640] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:33.815773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2e6c59-f6d6-40e6-b7ed-ebc4275b723b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.841360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:33.876149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214cfb7-6856-4f4a-8312-7999b326070d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.924642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2e6c59-f6d6-40e6-b7ed-ebc4275b723b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:33.925131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.925948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.926613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.927060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.941085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:33.945983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:33.947162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:33.948738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:33.954915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 319/641] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:33.954915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:33.976950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:33.977813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:33.978254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:33.978659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:33.980031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:33.980479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:33.982516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:33.998932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 323/642] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:34.000058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.066193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60c4d74-a397-4ba3-a649-6e200240ae27 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.088625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.104214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.105495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.105773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.106036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.112828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.114149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.115742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:34.127823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:34.128973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:34.129904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 320/643] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:34.130419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.132895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd5d2d-4bd5-4cc7-8104-4e449686000b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.170631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.171260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.171544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.171810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.172470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:34.172772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.174141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.182112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 324/644] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:34.186806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.243024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19149d89-0f9c-44e5-a664-be77cb279041 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.278090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.278701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.279002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.279273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.287830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.289754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.298491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d29265-1330-4e61-8e97-af8cd463940f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:34.302430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:34.303598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:19:34.304706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:34.308296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 321/645] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/users/cinder => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:34.309243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.336786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.337415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.337693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.338139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.348823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:34.352372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:34.354042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:34.356142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 325/646] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:34.357012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.367863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.419343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50eeb8e-95c7-46b6-80db-7ab2ce1bbf8f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.454341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.454930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.455231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.455491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.456172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:34.456470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.457874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.465141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91c3401-b98e-4e4b-9a5b-db758a514ed7 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.471198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 322/647] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:34.476109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.503664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.504367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.504706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.505043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.516437 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.517950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:34.532440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:34.533562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:34.534669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:34.534669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:34.534669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:34.535303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 326/648] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:34.535899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.580385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.585111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8b212f-735e-4c4b-9f12-f2db5a6b16ee None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.598036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.622874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.623503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.623778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.624045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.624758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:34.625059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.626470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.640914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb844c9-d941-45ff-9666-81fcd6593c55 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.650534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 323/649] 10.222.0.33 () {56 vars in 1137 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:34.650534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.679497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.680097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.680369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.680632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.681277 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:34.681571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.683171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.695275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 327/650] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:34.696702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.756055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3630af16-f1dd-474b-81a6-6fe53f4d18c9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.791822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.793022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.793294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.793556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.794272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=c8d2db883de44964b6e15b85a97726b5) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:34.794569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.796094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.799803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b3f6b-db4f-43db-8ab9-d0eaf7f8ed91 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.802018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 324/651] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/role_assignments?user.id=c8d2db883de44964b6e15b85a97726b5&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:34.802794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.834899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.842175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:34.858781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.859929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.860404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.860846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:34.861972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:34.862443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:34.864487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:34.881809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 328/652] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:34.883088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:34.950205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90978d0-41a4-4766-8c75-6c2567e0aef7 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:34.990880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:34.991919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:34.992392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:34.992859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.015941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265c4c3b-c6c9-479e-9617-4a25fef6109d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.054794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.055490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.055796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.056145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.070762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.071909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.072110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.075179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:35.089412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:35.090422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:35.091449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:35.091449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:35.091449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:35.091449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 329/653] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:35.091697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.104845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.135307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.194769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 325/654] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:35.195706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.230809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62227db2-3a02-4131-8367-6ff64a22134b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.272907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.273545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.273820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.274085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.283431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.284992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:35.297974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:35.299666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:35.300826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 330/655] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/users/neutron => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:35.303906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.344700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33414ba-3e85-441e-a188-b8be5d5eeb77 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.376451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.390145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.390677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.390953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.391264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.391903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:35.392204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.393549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.406624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 326/656] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:35.407473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.413269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.422341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28b9dce-e5be-4c0b-8126-2d1275fb68bb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.463261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a668cc-71bc-4c6a-b8bd-dd904606a268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.464860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a668cc-71bc-4c6a-b8bd-dd904606a268 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.465569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a668cc-71bc-4c6a-b8bd-dd904606a268 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.466013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a668cc-71bc-4c6a-b8bd-dd904606a268 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.477884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a668cc-71bc-4c6a-b8bd-dd904606a268 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.479958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a668cc-71bc-4c6a-b8bd-dd904606a268 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-76a668cc-71bc-4c6a-b8bd-dd904606a268 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:35.494290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:35.495758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:35.496688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:35.496688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:35.496688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:35.500406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 331/657] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:35.501470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ad21c-45ab-48ab-88ff-280e8175bb3e None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.501760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ad21c-45ab-48ab-88ff-280e8175bb3e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.502060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ad21c-45ab-48ab-88ff-280e8175bb3e None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.503468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 332/658] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:35.508161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6ad21c-45ab-48ab-88ff-280e8175bb3e None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.519544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039242bf-3d61-411f-a6de-8f7025341388 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.556496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.557096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.557370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.558227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.559071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:35.559382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.560841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.583740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 327/659] 10.222.0.33 () {56 vars in 1139 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:35.583740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.628278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6ad21c-45ab-48ab-88ff-280e8175bb3e None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.631996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.664510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e387a8a6-b354-45de-9d29-2b07b43af2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.665058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e387a8a6-b354-45de-9d29-2b07b43af2a0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.665337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e387a8a6-b354-45de-9d29-2b07b43af2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.665616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e387a8a6-b354-45de-9d29-2b07b43af2a0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.673755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e387a8a6-b354-45de-9d29-2b07b43af2a0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.675426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e387a8a6-b354-45de-9d29-2b07b43af2a0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-e387a8a6-b354-45de-9d29-2b07b43af2a0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:35.687227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:35.688415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:35.689376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 333/660] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/users/glance => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:35.689972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c93784-1d78-41ab-8ed7-b506877dcec1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.690048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c93784-1d78-41ab-8ed7-b506877dcec1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.690320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c93784-1d78-41ab-8ed7-b506877dcec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.697281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.700534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e638f1f0-4925-4232-85dd-b09009806718 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.709148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-12c93784-1d78-41ab-8ed7-b506877dcec1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:35.710344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-12c93784-1d78-41ab-8ed7-b506877dcec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:35.741574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.742278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.742617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.743023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.743620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:35.743919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.746086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.758283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 328/661] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:35.760309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.780978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 334/662] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:35.781853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c93784-1d78-41ab-8ed7-b506877dcec1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:35.884329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.888938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9b537-b30e-4874-92cd-60201a516fbe None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.915890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c93784-1d78-41ab-8ed7-b506877dcec1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:35.930655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.931298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.931605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.931855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.963512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:35.965297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:35.966221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:35.966865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:35.967900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:35.968639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:35.970925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:35.972962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:35.995503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 335/663] 10.222.0.33 () {56 vars in 1137 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:35.997278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.013149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.014750 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.121964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87d0d1e-3f82-43b2-9113-67013432b8be None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.127318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.135048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 329/664] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:36.135048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.165584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.166278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.166490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.166761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.174184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.175651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:36.187223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:36.188194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 336/665] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:36.189100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.244496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2e760c-ee06-4422-9fc7-01ffefed06c4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.253438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.284352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.284953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.285230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.285668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.304624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b38536-4bab-4edb-b757-6be831ae0006 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.340682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.341313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.341586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.341857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.352625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:36.356146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:36.358066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:36.363367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:36.364209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:36.364850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.364939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 337/666] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:36.364939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.365092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.465816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca0c8f0-71d9-485f-a93e-9e0ed2ed84e0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.469944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 330/667] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:36.471922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.479223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.502929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.503686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.503962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.504232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.515514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.516965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:36.534370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:36.535424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:36.536316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:36.536316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:36.536316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:36.536316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 338/668] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:36.536316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.584808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.588423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5f113d-7a96-4bde-b70a-70fa08892353 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.626730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.627354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.627630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.627898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.628593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:36.628878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.630288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.646664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 331/669] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:36.646664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.646933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b10f29-0c88-40b9-b2c7-7446173d04f7 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.678838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.679471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.679751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.680025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.680688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:36.680992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.682364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.691254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 339/670] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:36.692066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.720136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.769992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d607349-1517-40af-8917-274c9d30dabf None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.798286 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce23078-2f59-4ba6-9915-9b252a3149e4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.806765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.808230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.808230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.808561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.819417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.820867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.834269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.834875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.835197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.835471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:36.835573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:36.836462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:36.844141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:36.844141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:36.844141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:36.844450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:36.844450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:36.844600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 332/671] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:36.844600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.844726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:36.859906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.860340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 340/672] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:36.861066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:36.944336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:36.959043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc85f69-088f-4be3-b7b4-34942e48c783 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.960361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae66598-36d1-44ed-8d59-f08d59c594e8 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:36.995629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.996178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:36.996455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:36.996725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:36.998863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:36.999757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.000171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.000574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.001435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=61d1b78e81db4dbe987d788fd398e89d) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:37.001882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.003632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.007283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.008809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.011732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 333/673] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/role_assignments?user.id=61d1b78e81db4dbe987d788fd398e89d&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:37.013294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:37.027144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:37.028464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:37.029574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:37.029574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:37.029574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:37.033356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 341/674] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:37.034374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.113799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:37.146287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4e5733-5ba6-4b67-bf07-793bab2981a2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.153006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359ae06-e693-4c2e-a528-30c64a22c51a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.183811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.184742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.185025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.185302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.186306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:37.186609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.188016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.190320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4650a1c8-8d34-46ff-ad2b-d346c5ed22c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.190983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4650a1c8-8d34-46ff-ad2b-d346c5ed22c3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.191271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4650a1c8-8d34-46ff-ad2b-d346c5ed22c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.191547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4650a1c8-8d34-46ff-ad2b-d346c5ed22c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.199578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4650a1c8-8d34-46ff-ad2b-d346c5ed22c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.201045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4650a1c8-8d34-46ff-ad2b-d346c5ed22c3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.201552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 334/675] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:37.202457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.207936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-4650a1c8-8d34-46ff-ad2b-d346c5ed22c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:37.223614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:37.224714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:37.225660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 342/676] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:37.226759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d807d2-bf16-4aaa-a2f8-45191e3beaf7 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.227067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d807d2-bf16-4aaa-a2f8-45191e3beaf7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.227341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d807d2-bf16-4aaa-a2f8-45191e3beaf7 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.232624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 343/677] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:37.232624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d807d2-bf16-4aaa-a2f8-45191e3beaf7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.327996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc16aea2-0548-4cb5-aed4-01c5f7c4e1b9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.335780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d807d2-bf16-4aaa-a2f8-45191e3beaf7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.367179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.367757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.368034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.368305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.368939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:37.369234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.370305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11049812-b81e-4393-8175-8b293d6bd798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.370796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.370918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11049812-b81e-4393-8175-8b293d6bd798 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.371234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11049812-b81e-4393-8175-8b293d6bd798 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.371507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11049812-b81e-4393-8175-8b293d6bd798 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.373239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:37.378852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11049812-b81e-4393-8175-8b293d6bd798 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.380340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11049812-b81e-4393-8175-8b293d6bd798 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.388805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 335/678] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:37.388805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-11049812-b81e-4393-8175-8b293d6bd798 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:37.392464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:37.393357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:37.394269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 344/679] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:37.395093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31cb32f-a841-4c44-a230-1ae615d46444 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.395187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31cb32f-a841-4c44-a230-1ae615d46444 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.395418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31cb32f-a841-4c44-a230-1ae615d46444 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.413975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d31cb32f-a841-4c44-a230-1ae615d46444 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:37.415180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d31cb32f-a841-4c44-a230-1ae615d46444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:37.474417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:37.488826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 345/680] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:37.495644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31cb32f-a841-4c44-a230-1ae615d46444 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.515777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd6a072-df0d-4b8d-b6ce-a7ee8f3d15ef None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.552413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.553014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.553287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.553576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.554254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:37.554562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.556016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.569389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 336/681] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:37.570231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.634927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:37.644153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31cb32f-a841-4c44-a230-1ae615d46444 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.682623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bf7551-07e8-4208-9d56-25f23be788f1 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.692584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.693610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.693902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.694314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.705094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.706575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:37.722361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:37.723547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:37.724574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:37.725143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.725645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.725802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.729664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.729664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:37.729664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.730176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 346/682] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/users/swift => generated 83 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:37.730176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.730542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.744801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 337/683] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:37.746761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.781033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:37.835805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405a37db-faee-4a1d-ae6e-ab8f94477613 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.862019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00838862-5e11-42ba-9d2c-57abe3559c0c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:37.875121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.876058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.876562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.877146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.878166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=87829678318948f3b81bd107076bdfa7) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:37.878704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.880266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:37.880880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.888641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 347/684] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/role_assignments?user.id=87829678318948f3b81bd107076bdfa7&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:37.889734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:37.913681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:37.914639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:37.915224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:37.915682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:37.916624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:37.918351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:37.920428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:37.945237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 338/685] 10.222.0.33 () {56 vars in 1135 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:37.946379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.030787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f350ad-f19d-4d2e-a6b2-63d5afd3647d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.051200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:38.082891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3661a4-cbbe-40a5-b533-2765bf19a798 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.086080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.086952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.087332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.087608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.088341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=b34c61ebdb9147d5b3d66cbfa556f38e) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:38.088709 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.090258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.096762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 348/686] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/role_assignments?user.id=b34c61ebdb9147d5b3d66cbfa556f38e&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:38.097477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.126999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.127731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.128010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.128289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.142370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:38.147250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:38.148443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:38.149507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:38.153937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 339/687] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:38.155270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.206689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:38.243842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c37003f-b227-491f-9403-8fcba9cdaae0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.293795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.294708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e04253a-8937-448d-8ad1-e303f29cdac5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.294905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.295475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.296003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.351525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.352307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.352674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.353450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.354789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:38.355237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.357084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.361975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:38.369234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 340/688] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:38.376575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.390922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.392485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.507737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:38.515242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2223ed9d-c9a2-4d3d-98d2-58a4fb67a4a9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.522879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 349/689] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:38.529102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.553356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.553992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.554583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.554583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.566152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.567711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:38.583608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:38.584611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:38.585556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:38.585556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:38.585556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:38.588463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 341/690] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:38.590943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.627561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:38.666114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69a73f7-c8e7-463d-b6c3-e8f60ee93d8d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.707249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5e2c8f-75d9-4bb4-885e-6bc796d82383 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.715725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.716659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.717230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.717700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.729948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.732089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.743385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.743984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.744260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.744568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.745195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:38.745494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.746842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:38.750817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:38.751853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:38.752738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:38.752738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:38.752738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:38.754897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 350/691] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:38.760898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.763091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 342/692] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:38.765005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.776681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:38.872181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ef7839-2c4d-4a99-b05f-5281e9e238cc None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.880811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9310e4b-d6a2-47a1-b22f-4343ad7153af None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:38.909502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.909905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.910183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.910455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.914216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:38.919275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.920589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.926308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:38.927200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:38.927703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:38.928172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:38.929074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:38.929603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:38.931207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:38.933784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:38.934700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:38.935712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:38.939065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 343/693] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:38.940822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:38.949685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 351/694] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:38.956056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:39.012208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:39.059069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4151c217-203d-4362-9875-7d23bdfdf657 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:39.094452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:39.095105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.095343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.095615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.096406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:39.096712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:39.098126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:39.110624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82aa328-b6c5-433f-a174-2e031b3f6d09 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:39.123078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 344/695] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:39.125065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:39.154931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:39.155965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.156528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.157031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.165667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:39.167985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:39.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:39.182672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:39.183616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:39.186022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 352/696] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:39.191875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:39.257973 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:39.287204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619b50c-675f-46f4-9809-965a8067e315 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:39.328035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3ba2fd-0bc8-4a63-b59b-4719f855ace5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:39.341452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:39.360709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d279ebd4-88cb-4277-815d-43cc64bdfb0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:39.361295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d279ebd4-88cb-4277-815d-43cc64bdfb0e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.361580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d279ebd4-88cb-4277-815d-43cc64bdfb0e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.362103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d279ebd4-88cb-4277-815d-43cc64bdfb0e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.383847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:39.386307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.387019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.387766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.389155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:39.390061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:39.392602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:39.412953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 353/697] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:39.420208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:39.545119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d279ebd4-88cb-4277-815d-43cc64bdfb0e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:39.546466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d279ebd4-88cb-4277-815d-43cc64bdfb0e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:39.555344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd75a55-238f-45dc-9fec-e1b575afa6c3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:39.579380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:39.634337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:39.635160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.636138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.636507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.637389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=2e280c4b418e4aafb822a617424c7e11) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:39.638028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:39.639610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:39.652331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 354/698] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:39 2022] GET /identity/v3/role_assignments?user.id=2e280c4b418e4aafb822a617424c7e11&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 608 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:39.726179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 345/699] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 602 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:39.732696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:39.762763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0034ac-1049-48bd-84b3-d52c1716d9fa None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.763319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0034ac-1049-48bd-84b3-d52c1716d9fa None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.763734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0034ac-1049-48bd-84b3-d52c1716d9fa None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.765298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 346/700] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:39.771214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf601122-db24-4844-b7dd-73e7f0c01d21 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.771720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf601122-db24-4844-b7dd-73e7f0c01d21 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.772130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf601122-db24-4844-b7dd-73e7f0c01d21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.813659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bf601122-db24-4844-b7dd-73e7f0c01d21 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:39.815338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bf601122-db24-4844-b7dd-73e7f0c01d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:39.846568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:39.894530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9b9c58-27d1-41da-936b-7b04dd22ab77 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:39.920094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 347/701] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:39.945923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:39.946885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:39.947348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:39.947765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:39.961701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:39.963432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:39.980947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:39.982117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:39.983288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:39.983288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:39.983288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:39.984361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 355/702] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:39.990305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf601122-db24-4844-b7dd-73e7f0c01d21 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:40.093608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf601122-db24-4844-b7dd-73e7f0c01d21 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:40.132135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:40.133497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.134015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.134462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.135297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:40.136334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:40.137860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:40.150824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 348/703] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:40.157259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:40.201320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:40.280944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:40.302987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19922af8-59ab-4822-9308-8270ca7c9842 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:40.311320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:40.341733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:40.342363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.342807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.343114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.353501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:40.353501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:40.363712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:40.364681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:40.365606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 356/704] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:40.371281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:40.418835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b520834a-9c49-4f68-a68d-fa69d790cfad None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:40.457572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:40.458567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.459288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.460172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.461255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:40.462814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:40.478538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:40.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:40.481212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:40.483839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 349/705] 10.222.0.33 () {56 vars in 1129 bytes} [Thu Oct 6 04:19:40 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:40.492498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:40.495570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:40.533483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4efdcd-14b4-4fa8-99bf-acc55c357103 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:40.591335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:40.611467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3169121b-92fa-4b76-bf13-a53ab113b7b8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:40.648116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:40.649078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.649495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.649917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.650678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:40.651740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:40.653446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:40.657818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:40.658450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.658730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.659028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.659624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 350/706] 10.222.0.33 () {56 vars in 1124 bytes} [Thu Oct 6 04:19:40 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:40.659821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:40.660131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:40.661655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:40.669544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:40.686338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 357/707] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:40.696078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:40.790693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdede66c-ce59-4f71-9438-0ad35837da19 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:40.830801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f080fa2b-763b-493d-ba35-94e5699e0009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:40.831785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f080fa2b-763b-493d-ba35-94e5699e0009 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.832839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f080fa2b-763b-493d-ba35-94e5699e0009 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.833309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f080fa2b-763b-493d-ba35-94e5699e0009 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.834127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f080fa2b-763b-493d-ba35-94e5699e0009 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:40.835109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f080fa2b-763b-493d-ba35-94e5699e0009 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:40.837321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f080fa2b-763b-493d-ba35-94e5699e0009 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:40.845417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 351/708] 10.222.0.33 () {56 vars in 1124 bytes} [Thu Oct 6 04:19:40 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:40.847519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951e51f-487f-4221-9333-09fa88a618e7 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.848049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951e51f-487f-4221-9333-09fa88a618e7 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.848156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e951e51f-487f-4221-9333-09fa88a618e7 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.855125 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 352/709] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:40.856211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da00a1-19fe-48bf-ac74-b56fb75e0bb9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.856666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da00a1-19fe-48bf-ac74-b56fb75e0bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.857061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da00a1-19fe-48bf-ac74-b56fb75e0bb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.876781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a266545-7f34-408d-84dd-ce612a97b2fe None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:40.882163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-11da00a1-19fe-48bf-ac74-b56fb75e0bb9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:40.883433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-11da00a1-19fe-48bf-ac74-b56fb75e0bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:40.886381 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:40.919838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a79e23-5009-43a9-b2bc-20e04cfb6acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:40.920840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a79e23-5009-43a9-b2bc-20e04cfb6acc None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:40.921333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a79e23-5009-43a9-b2bc-20e04cfb6acc None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:40.921859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a79e23-5009-43a9-b2bc-20e04cfb6acc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:40.922779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a79e23-5009-43a9-b2bc-20e04cfb6acc None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=c8d2db883de44964b6e15b85a97726b5) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:40.923316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a79e23-5009-43a9-b2bc-20e04cfb6acc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:40.925205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a79e23-5009-43a9-b2bc-20e04cfb6acc None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:40.932005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 358/710] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:40 2022] GET /identity/v3/role_assignments?user.id=c8d2db883de44964b6e15b85a97726b5&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:40.961354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 353/711] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:41.139155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8106967-945e-489c-aa87-71b80af848a2 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:41.139806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8106967-945e-489c-aa87-71b80af848a2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:41.140282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8106967-945e-489c-aa87-71b80af848a2 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:41.141938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 359/712] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:41.157682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd530437-0b51-4ec4-bcf7-810747abb107 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:41.158243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd530437-0b51-4ec4-bcf7-810747abb107 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:41.158243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd530437-0b51-4ec4-bcf7-810747abb107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:41.185082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bd530437-0b51-4ec4-bcf7-810747abb107 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:41.186672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bd530437-0b51-4ec4-bcf7-810747abb107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:41.273872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 354/713] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:41.336659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8106967-945e-489c-aa87-71b80af848a2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:41.348646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:41.484668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8106967-945e-489c-aa87-71b80af848a2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:41.530794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:41.531485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:41.531721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:41.531988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] PATH_INFO: `/v3/services/network` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:41.532567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:41.533911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:41.548850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:41.549914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:41.550848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 360/714] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:41 2022] GET /identity/v3/services/network => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:41.558777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd530437-0b51-4ec4-bcf7-810747abb107 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:41.664797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:41.702402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:41.706943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd530437-0b51-4ec4-bcf7-810747abb107 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:41.742016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:41.742534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:41.742811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:41.743114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:41.743829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] RBAC: Adding query filter params (name=network) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:41.744111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:41.746080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:41.750715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 355/715] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:19:41 2022] GET /identity/v3/services?name=network => generated 124 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:41.760932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:41.857743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:41.920428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670f2f73-f2f5-43ef-8588-6e409e1ae965 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:41.952842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d7d3fa-54b5-41ca-87cb-4b1833610087 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:41.990577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:41.995872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:41.996435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:41.996713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:41.996977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:41.997299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:41.998097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:41.998573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:41.999191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:41.999997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] RBAC: Adding query filter params (type=network) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:42.000500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:42.002037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:42.007516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 356/716] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:19:41 2022] GET /identity/v3/services?type=network => generated 124 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:42.028569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:42.030073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:42.052467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:42.053493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:42.054402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:42.054402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:42.054402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:42.054402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 361/717] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:42.060206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:42.188298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9457222a-5042-4055-87be-041e294fb7a1 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:42.230602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f59480-12e9-478c-8607-dfa4f2e439d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:42.231258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f59480-12e9-478c-8607-dfa4f2e439d1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:42.231890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f59480-12e9-478c-8607-dfa4f2e439d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:42.232166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f59480-12e9-478c-8607-dfa4f2e439d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:42.232825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f59480-12e9-478c-8607-dfa4f2e439d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:42.233127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f59480-12e9-478c-8607-dfa4f2e439d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:42.234513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f59480-12e9-478c-8607-dfa4f2e439d1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:42.247962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 357/718] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:42.256395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:42.382318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80435c8c-33b0-4d42-8364-5ca8c358b44f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:42.414336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:42.429049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9c6cc3-9a73-4e08-ba72-624bdf597902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:42.429657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9c6cc3-9a73-4e08-ba72-624bdf597902 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:42.429935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9c6cc3-9a73-4e08-ba72-624bdf597902 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:42.430213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9c6cc3-9a73-4e08-ba72-624bdf597902 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:42.430864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9c6cc3-9a73-4e08-ba72-624bdf597902 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:42.432386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9c6cc3-9a73-4e08-ba72-624bdf597902 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:42.472120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 362/719] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:19:42 2022] POST /identity/v3/users => generated 254 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:42.629322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ab20a-b935-4d2c-88dc-e24d5462f904 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:42.629968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ab20a-b935-4d2c-88dc-e24d5462f904 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:42.630436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ab20a-b935-4d2c-88dc-e24d5462f904 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:42.632297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 358/720] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:42.638221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f156fbe2-de57-41d7-a11f-fe977ea375c2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:42.638700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f156fbe2-de57-41d7-a11f-fe977ea375c2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:42.638794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f156fbe2-de57-41d7-a11f-fe977ea375c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:42.668341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f156fbe2-de57-41d7-a11f-fe977ea375c2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:42.669588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f156fbe2-de57-41d7-a11f-fe977ea375c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:42.754066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 363/721] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:43.049386 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110ab20a-b935-4d2c-88dc-e24d5462f904 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:43.117852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:43.182680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110ab20a-b935-4d2c-88dc-e24d5462f904 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:43.240453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:43.241089 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.241357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.241760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.267928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:43.273132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:43.274077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:43.274932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 359/722] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:43.281081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f156fbe2-de57-41d7-a11f-fe977ea375c2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:43.439026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f156fbe2-de57-41d7-a11f-fe977ea375c2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:43.455250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:43.497732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8c5a2b-a71f-45b5-b5c8-239e753fa83b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:43.498340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8c5a2b-a71f-45b5-b5c8-239e753fa83b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.498614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8c5a2b-a71f-45b5-b5c8-239e753fa83b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.498879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8c5a2b-a71f-45b5-b5c8-239e753fa83b None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.499583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8c5a2b-a71f-45b5-b5c8-239e753fa83b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:43.499879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8c5a2b-a71f-45b5-b5c8-239e753fa83b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:43.501236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8c5a2b-a71f-45b5-b5c8-239e753fa83b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:43.515216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 364/723] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:43.520869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:43.596095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9374a4ad-d953-4a4a-894a-78a63f8002db None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.596434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9374a4ad-d953-4a4a-894a-78a63f8002db None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.596651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9374a4ad-d953-4a4a-894a-78a63f8002db None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.598017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 365/724] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:19:43.612846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35ef7f-58d5-4f54-baa0-a623b10bf62c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.613177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35ef7f-58d5-4f54-baa0-a623b10bf62c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.613397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a35ef7f-58d5-4f54-baa0-a623b10bf62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.630947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0a702f-4a6f-4cb1-b608-831388d86149 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:43.656026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-8a35ef7f-58d5-4f54-baa0-a623b10bf62c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:43.657238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8a35ef7f-58d5-4f54-baa0-a623b10bf62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:43.659319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:43.671649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:43.672277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.672555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.672826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.691637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:43.693016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:43.717978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:43.719134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:43.720054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:43.720054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:43.720054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:43.720646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 360/725] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:43.730716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:43.788891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 366/726] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:43.790486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1478861e-1ab3-4f1d-b4d0-7e3cb3af9a96 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.790776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1478861e-1ab3-4f1d-b4d0-7e3cb3af9a96 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.792180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1478861e-1ab3-4f1d-b4d0-7e3cb3af9a96 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.793582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 367/727] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:43.799162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5abd76-525f-49f8-b4fc-2239374aa4eb None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.799538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5abd76-525f-49f8-b4fc-2239374aa4eb None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.799885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5abd76-525f-49f8-b4fc-2239374aa4eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.824646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-8d5abd76-525f-49f8-b4fc-2239374aa4eb None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:43.826670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8d5abd76-525f-49f8-b4fc-2239374aa4eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:43.862121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d36bb8-567b-487e-8f43-f1fb2bc9aeae None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:43.929620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:43.930233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:43.930511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:43.930784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:43.931447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:43.931755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:43.933535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:43.956757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 361/728] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:43.959124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 368/729] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 06 04:19:43.973925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.008247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:44.098169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5abd76-525f-49f8-b4fc-2239374aa4eb None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.101938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adb22b-7334-4140-9f11-0b0f0c71285b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:44.140904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:44.141539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.141811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.142079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.150286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:44.152285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:44.166018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:44.167404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:44.168383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 362/730] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:43 2022] GET /identity/v3/users/swift => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:44.174220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.181202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:44.284989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5abd76-525f-49f8-b4fc-2239374aa4eb None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:44.291442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:44.299631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253bcbcb-3e52-4e3a-abd7-c0e7d87256a5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:44.335345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:44.336056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.336340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.336617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.337341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:44.337648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:44.339857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:44.350424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:44.351120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.351402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.351671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.363139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 363/731] 10.222.0.33 () {56 vars in 1135 bytes} [Thu Oct 6 04:19:44 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:44.363413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:44.364108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:44.368806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:44.369801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:44.370726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 369/732] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:44.371483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.472355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:44.475202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0131e07-d159-4556-b3c7-9e25dfeba31b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:44.482213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e939b0-9ae5-4435-b23e-e8fbeab22e69 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:44.522983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:44.522983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.522983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.522983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.523453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:44.523539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bb889b-2e72-45b6-966a-74ae4050daad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:44.523635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:44.523719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bb889b-2e72-45b6-966a-74ae4050daad None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.523719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bb889b-2e72-45b6-966a-74ae4050daad None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.523719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bb889b-2e72-45b6-966a-74ae4050daad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.539937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 364/733] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:19:44 2022] POST /identity/v3/services => generated 251 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:44.540944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.566121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:44.704410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780a3f48-c9eb-434b-afc8-8616229befd1 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:44.710690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bb889b-2e72-45b6-966a-74ae4050daad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:44.713135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bb889b-2e72-45b6-966a-74ae4050daad None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:44.742837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c10e5c7-6451-4cf0-aef2-07fd7cef6205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:44.743392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10e5c7-6451-4cf0-aef2-07fd7cef6205 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.743671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10e5c7-6451-4cf0-aef2-07fd7cef6205 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.743943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10e5c7-6451-4cf0-aef2-07fd7cef6205 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.745124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c10e5c7-6451-4cf0-aef2-07fd7cef6205 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:44.745427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c10e5c7-6451-4cf0-aef2-07fd7cef6205 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:44.746925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c10e5c7-6451-4cf0-aef2-07fd7cef6205 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:44.757794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 365/734] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:44.759098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b792ade-36ce-4d7e-84b5-1d48af77e21f None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.759385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b792ade-36ce-4d7e-84b5-1d48af77e21f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.759656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b792ade-36ce-4d7e-84b5-1d48af77e21f None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.762123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 366/735] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:44.762713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b792ade-36ce-4d7e-84b5-1d48af77e21f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.833275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:44.850472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 370/736] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 06 04:19:44.852433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ea8711-cf7e-40cd-a862-51eebd40edb8 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.853091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ea8711-cf7e-40cd-a862-51eebd40edb8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.853468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ea8711-cf7e-40cd-a862-51eebd40edb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.855385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:44.877680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-48ea8711-cf7e-40cd-a862-51eebd40edb8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:44.879999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-48ea8711-cf7e-40cd-a862-51eebd40edb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:44.900859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b792ade-36ce-4d7e-84b5-1d48af77e21f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:44.946524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:44.947187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:44.947462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:44.948322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:44.959577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:44.961090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:44.974681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 371/737] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:44 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 06 04:19:44.975999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ea8711-cf7e-40cd-a862-51eebd40edb8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:44.977801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:44.978759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:44.980010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:44.980010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:44.980010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:44.983075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 367/738] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:44.985168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.098342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df20c007-c270-4408-b6ec-5da9a469da71 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.106988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ea8711-cf7e-40cd-a862-51eebd40edb8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.120561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:45.141314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:45.161207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.161836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.162116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.162385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.164521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.165111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.165380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.165642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.166274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:45.167182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.168707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.187168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 368/739] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:45.188251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.189915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.191988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:45.233870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:45.234852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:45.235791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:45.235791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:45.235791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:45.244862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 372/740] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:45.245962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.380781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7557eae7-c969-4e9a-be63-00b9cc6fdcdb None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.409126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f388c4e-7cf8-48ff-ae91-f885d303d5fb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.419034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.419805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.420080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.420347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.428254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.429927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:45.442372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:45.443413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:45.444380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:45.447440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 369/741] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:19:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:45.448815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.451728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.452564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.453033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.453703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.454634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:45.455157 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.456957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.471387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 373/742] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:45.472533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.491135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:45.512311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:45.580691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bd04d-065d-4e05-bb5b-30af57aa9432 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.605477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59169833-9fa3-44f6-9b7b-393d2dc3814d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.626104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.626607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.626884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.627186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.627832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.629237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.644663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 370/743] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:19:45 2022] POST /identity/v3/services => generated 240 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:45.644963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0944a26-7491-46d8-90a8-e7615f9fbd29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.645386 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.646122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0944a26-7491-46d8-90a8-e7615f9fbd29 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.646590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0944a26-7491-46d8-90a8-e7615f9fbd29 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.647160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0944a26-7491-46d8-90a8-e7615f9fbd29 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.648141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0944a26-7491-46d8-90a8-e7615f9fbd29 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:45.648673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0944a26-7491-46d8-90a8-e7615f9fbd29 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.650252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0944a26-7491-46d8-90a8-e7615f9fbd29 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.675016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 374/744] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:19:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:45.675357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d7f00c-7846-4a5d-84b9-620d44e39aa9 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.675635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d7f00c-7846-4a5d-84b9-620d44e39aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.675908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d7f00c-7846-4a5d-84b9-620d44e39aa9 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.677558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 375/745] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:45.679252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d7f00c-7846-4a5d-84b9-620d44e39aa9 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.779532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:45.794932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:45.798351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f4bc4-24a9-40a8-83df-c550952748df None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.807810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d7f00c-7846-4a5d-84b9-620d44e39aa9 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:45.847353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea7e89f-dfea-4aa9-84fd-8ffcc666ebcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.848284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea7e89f-dfea-4aa9-84fd-8ffcc666ebcc None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.848737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea7e89f-dfea-4aa9-84fd-8ffcc666ebcc None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.849010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea7e89f-dfea-4aa9-84fd-8ffcc666ebcc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.851719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:45.852357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.852625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.852900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.857219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea7e89f-dfea-4aa9-84fd-8ffcc666ebcc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.858655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea7e89f-dfea-4aa9-84fd-8ffcc666ebcc None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.870104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:45.871902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-dea7e89f-dfea-4aa9-84fd-8ffcc666ebcc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:45.872483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:45.873374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:45.874293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 371/746] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:45.876859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6807441d-a36b-43a5-b11b-08597870d08d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:45.876859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6807441d-a36b-43a5-b11b-08597870d08d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:45.876859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6807441d-a36b-43a5-b11b-08597870d08d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:45.894086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:45.895474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:45.896409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:45.896409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:45.896409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:45.897576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6807441d-a36b-43a5-b11b-08597870d08d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:45.898752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6807441d-a36b-43a5-b11b-08597870d08d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:45.899863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 376/747] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:45.899863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:45.981042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 372/748] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:45 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:45.986983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6807441d-a36b-43a5-b11b-08597870d08d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:46.064148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153a6e6a-d4e7-4eb5-9b9b-c567677bc9d4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.070974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:46.085796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6807441d-a36b-43a5-b11b-08597870d08d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.121993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.122590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.122863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.123182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.123809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:46.124103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:46.125459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:46.136719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.137325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.137607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.137974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.138131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 373/749] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:46.138558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:46.138859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:46.140332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:46.143059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:46.171081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 377/750] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:46.186905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:46.204135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:46.275276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22d976d-2bbf-4953-a8d5-e9190bd1814a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.323023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc33942-8a04-49ce-a01e-43784e40d25a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.337061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.337995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.339217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.339726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.348221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:46.350400 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:46.357896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.358468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.358739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.359038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.359762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=2e280c4b418e4aafb822a617424c7e11) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:46.360066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:46.361470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:46.365749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:46.366658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:46.367613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:46.368052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 378/751] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:46 2022] GET /identity/v3/role_assignments?user.id=2e280c4b418e4aafb822a617424c7e11&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:46.368579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:46.370740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 374/752] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:46.381136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:46.386336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:46.492796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:46.555090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e53dcf-f1c4-4dfd-b15b-f4465fdd1465 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.580219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112281f-77a7-479d-ae69-114237ae029c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.606811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.607478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.607753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.608019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.617116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.617711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.617984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.618252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.619019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:46.619329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:46.620934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:46.625720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:46.631310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:46.632265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:46.633131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:46.639754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 375/753] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:46.641698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 379/754] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:46.643408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:46.647961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:46.711334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:46.752496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbc4e62-7d26-487b-9b7a-47a1bbc1e8db None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.789441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.790157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.790437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.790709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/c8d2db883de44964b6e15b85a97726b5/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.793270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=c8d2db883de44964b6e15b85a97726b5, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:46.794686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:46.810205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 376/755] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:46 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/c8d2db883de44964b6e15b85a97726b5/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:46.899132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52168fe-bc9f-4272-949b-cc4c16eea249 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:46.943449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:46.953414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc50aaac-49f3-4cb1-9eaa-08b159f41bac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:46.954038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc50aaac-49f3-4cb1-9eaa-08b159f41bac None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:46.954317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc50aaac-49f3-4cb1-9eaa-08b159f41bac None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:46.954588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc50aaac-49f3-4cb1-9eaa-08b159f41bac None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:46.955299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc50aaac-49f3-4cb1-9eaa-08b159f41bac None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:46.955602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc50aaac-49f3-4cb1-9eaa-08b159f41bac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:46.957016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc50aaac-49f3-4cb1-9eaa-08b159f41bac None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:46.971945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 380/756] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:46.978017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:47.095293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b12c7-f22e-42bf-9256-7aa509ff1fb0 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:47.095727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b12c7-f22e-42bf-9256-7aa509ff1fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:47.095941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b12c7-f22e-42bf-9256-7aa509ff1fb0 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:47.097412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 381/757] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:47.097941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da66e70-b3a6-4f37-a912-bcd0ed3d2607 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:47.106600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3f52a-3866-4b8a-b0af-052a4739e6dd None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:47.106600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3f52a-3866-4b8a-b0af-052a4739e6dd None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:47.106600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a3f52a-3866-4b8a-b0af-052a4739e6dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:47.132251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f8a3f52a-3866-4b8a-b0af-052a4739e6dd None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:47.133797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f8a3f52a-3866-4b8a-b0af-052a4739e6dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:47.134107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:47.134736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:47.135063 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:47.135397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:47.146444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:47.149171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:47.150625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:47.169649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:47.170690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:47.172768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:47.172768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:47.172768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:47.175386 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 377/758] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:47.176568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:47.232798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 382/759] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:47 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:19:47.327901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6a561-5e4c-496e-b3b3-00d59d40cb1c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:47.378142 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:47.379131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:47.379631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:47.380124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:47.380965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:47.381543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:47.383595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:47.404354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 378/760] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:47.409496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3f52a-3866-4b8a-b0af-052a4739e6dd None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:47.529306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:47.531565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a3f52a-3866-4b8a-b0af-052a4739e6dd None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:47.579236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:47.598017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:47.598662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:47.598939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:47.599230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:47.619861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:47.621325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:47.640654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:47.641656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:19:47.642614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:47.643186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 383/761] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:19:47 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:47.643756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:47.654034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:47.688919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf2af15-8037-4159-b1ca-74ae20dba303 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:47.770529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6d1747-3d6f-4f68-a396-4e23eebbdc87 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:47.776990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:47.777911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:47.778390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:47.778858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:47.779727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:47.781339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:47.800955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:47.802004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:47.802871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:47.805933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 379/762] 10.222.0.33 () {56 vars in 1129 bytes} [Thu Oct 6 04:19:47 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:47.813660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:47.814283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:47.814561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:47.814831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:47.815595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:47.815819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:47.815943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:47.817521 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:47.846729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 384/763] 10.222.0.33 () {56 vars in 1149 bytes} [Thu Oct 6 04:19:47 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:47.852198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:47.906696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:47.977222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:47.981171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd5fcc9-7e68-4532-9158-c772800331a4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:47.989074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2b55e-3991-4cf8-a662-3d1a7afce286 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:48.021551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:48.022155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.022432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.022702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.055792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:48.059814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.060092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.060371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.061010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:48.061312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:48.062704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:48.073962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 380/764] 10.222.0.33 () {56 vars in 1124 bytes} [Thu Oct 6 04:19:47 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:48.080009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:48.163540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:48.164914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:48.239382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:48.289123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c3e216-ffcf-41c1-a80c-06b1294950eb None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:48.331267 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:48.331874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.332149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.332417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.333044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:48.333403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:48.334769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:48.344649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 381/765] 10.222.0.33 () {56 vars in 1124 bytes} [Thu Oct 6 04:19:48 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:48.352721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:48.357466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 385/766] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:47 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 06 04:19:48.366606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:48.395083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:48.484753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0660909a-7f97-4e90-956d-4d80047bdc5c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:48.495102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c99ac26-762b-4344-a41c-2c67a6bd0782 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:48.532199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef51bed-be9e-412b-8789-ebac808f1583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:48.532413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4c5843-f765-43f2-89fa-173e3e10ad68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:48.532823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef51bed-be9e-412b-8789-ebac808f1583 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.532931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c5843-f765-43f2-89fa-173e3e10ad68 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.533089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef51bed-be9e-412b-8789-ebac808f1583 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.533181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c5843-f765-43f2-89fa-173e3e10ad68 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.533342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef51bed-be9e-412b-8789-ebac808f1583 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.533508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4c5843-f765-43f2-89fa-173e3e10ad68 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.534147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4c5843-f765-43f2-89fa-173e3e10ad68 None admin] RBAC: Adding query filter params (interface=public, service_id=3c491306d819425195aef130c7c8cfba, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:48.534450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4c5843-f765-43f2-89fa-173e3e10ad68 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:48.536560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4c5843-f765-43f2-89fa-173e3e10ad68 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:48.540791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:48.545773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef51bed-be9e-412b-8789-ebac808f1583 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:48.547366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef51bed-be9e-412b-8789-ebac808f1583 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:48.548169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 382/767] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:19:48 2022] GET /identity/v3/endpoints?service_id=3c491306d819425195aef130c7c8cfba&interface=public®ion_id=RegionOne => generated 194 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:48.549530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b3e9fc-a158-43f0-9ea0-89b36369fcc0 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.550032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b3e9fc-a158-43f0-9ea0-89b36369fcc0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.550474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b3e9fc-a158-43f0-9ea0-89b36369fcc0 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.552294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 383/768] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:48.553362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b3e9fc-a158-43f0-9ea0-89b36369fcc0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-fef51bed-be9e-412b-8789-ebac808f1583 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:48.572983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:48.574205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:48.575230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:48.575230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:48.575230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:48.578726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 386/769] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:48.579958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa29142-3ec2-4efb-8735-f56c7044d6f6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.580237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa29142-3ec2-4efb-8735-f56c7044d6f6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.580509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa29142-3ec2-4efb-8735-f56c7044d6f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.598922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-0fa29142-3ec2-4efb-8735-f56c7044d6f6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:48.600153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0fa29142-3ec2-4efb-8735-f56c7044d6f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:48.676646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:48.685053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 387/770] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:48.691856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa29142-3ec2-4efb-8735-f56c7044d6f6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:48.692586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b3e9fc-a158-43f0-9ea0-89b36369fcc0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:48.765820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473a0364-d71c-4c6c-96f8-2ffb709fd943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:48.770758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a0364-d71c-4c6c-96f8-2ffb709fd943 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.771327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a0364-d71c-4c6c-96f8-2ffb709fd943 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.771747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a0364-d71c-4c6c-96f8-2ffb709fd943 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.772522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473a0364-d71c-4c6c-96f8-2ffb709fd943 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:48.774162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473a0364-d71c-4c6c-96f8-2ffb709fd943 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:48.789148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 384/771] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:19:48 2022] GET /identity/v3/services => generated 772 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:48.871165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa29142-3ec2-4efb-8735-f56c7044d6f6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:48.899304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:48.899819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cdfa2d-216a-4f5e-85f9-c6dde32ea3dd None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.900719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cdfa2d-216a-4f5e-85f9-c6dde32ea3dd None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.900719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cdfa2d-216a-4f5e-85f9-c6dde32ea3dd None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.905904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 385/772] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:19:48.908536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caa524-ea96-4e35-a273-1e287f283027 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.908816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caa524-ea96-4e35-a273-1e287f283027 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.909086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caa524-ea96-4e35-a273-1e287f283027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.920600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:48.922141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:48.922564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:48.923008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:48.924508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:48.924991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:48.926837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:48.930123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f6caa524-ea96-4e35-a273-1e287f283027 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:48.931372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f6caa524-ea96-4e35-a273-1e287f283027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:48.942491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 388/773] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:48.949102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.077373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 386/774] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:19:49.078370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caa524-ea96-4e35-a273-1e287f283027 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.105718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3299438e-945a-478d-82ab-a26de896b909 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.144996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.145684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.145967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.146241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.155185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:49.157081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:49.174445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:49.177151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:49.178026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:49.179053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 389/775] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:49.185455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.201997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caa524-ea96-4e35-a273-1e287f283027 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.239320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:49.240841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.241973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.242656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.243223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] PATH_INFO: `/v3/services/network` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.244109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:49.246311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:49.256003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:49.257014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:49.257882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:49.260683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 387/776] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/services/network => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:49.266455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.376126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced2abe6-d0ed-4a9d-8623-d35de40cc86f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.396185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca8fd82-55fc-4567-b4ab-1496b0c64ff6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.413131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.414436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.414907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.415411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.416520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] RBAC: Adding query filter params (name=network) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:49.417019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:49.418610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:49.424214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 388/777] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/services?name=network => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:49.429546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.439375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.439985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.440260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.440530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.441221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:49.442082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:49.443623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:49.448924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:49.466928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 390/778] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:49.466928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.551688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ef2190-d0af-42ba-befb-11e7d21db64c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.577212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:49.581059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e61c78b-f656-4f5a-b876-912f090e46c5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.589679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.590314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.590642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.590914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.604660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:49.610189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:49.611281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:49.612439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 389/779] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:49.613145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.622988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e3effd-9619-430d-b9ec-3e38925b8495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.623731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e3effd-9619-430d-b9ec-3e38925b8495 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.624111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e3effd-9619-430d-b9ec-3e38925b8495 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.624488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e3effd-9619-430d-b9ec-3e38925b8495 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.625228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e3effd-9619-430d-b9ec-3e38925b8495 None admin] RBAC: Adding query filter params (type=network) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:49.625629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e3effd-9619-430d-b9ec-3e38925b8495 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:49.627190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e3effd-9619-430d-b9ec-3e38925b8495 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:49.637970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 391/780] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/services?type=network => generated 350 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:49.637970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e0306-82c4-45d7-bf92-76b6a5fd30f8 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.637970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e0306-82c4-45d7-bf92-76b6a5fd30f8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.637970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e0306-82c4-45d7-bf92-76b6a5fd30f8 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.637970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 392/781] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:49.637970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784e0306-82c4-45d7-bf92-76b6a5fd30f8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.699766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:49.732469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee321332-86af-40dd-9d9c-80cf0f96df97 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.744601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784e0306-82c4-45d7-bf92-76b6a5fd30f8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:49.773261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.773861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.774216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.774465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.775141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=371190691e284514bafcba1bdd199fc6) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:49.775449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:49.777067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:49.781849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0cec43-43f6-4c0c-8674-042345f5530e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:49.791872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0cec43-43f6-4c0c-8674-042345f5530e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.792076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 390/782] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/role_assignments?user.id=371190691e284514bafcba1bdd199fc6&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:49.792183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0cec43-43f6-4c0c-8674-042345f5530e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.792183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0cec43-43f6-4c0c-8674-042345f5530e None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.792317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.792387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0cec43-43f6-4c0c-8674-042345f5530e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:49.792387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0cec43-43f6-4c0c-8674-042345f5530e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:49.792387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0cec43-43f6-4c0c-8674-042345f5530e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:49.802056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 393/783] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:49.803197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0a8d5-5afb-4790-a308-61cf4d30d752 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:49.803482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0a8d5-5afb-4790-a308-61cf4d30d752 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:49.803822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0a8d5-5afb-4790-a308-61cf4d30d752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:49.824916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-72d0a8d5-5afb-4790-a308-61cf4d30d752 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:49.826139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-72d0a8d5-5afb-4790-a308-61cf4d30d752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:49.851470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:49.915074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 394/784] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:49.915074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0a8d5-5afb-4790-a308-61cf4d30d752 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:49.985647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31488a7b-442f-4372-8673-72e221e497ef None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.029337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.030099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.030618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.031077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.031871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] RBAC: Adding query filter params (interface=public, service_id=1f33246da51d433c8cf3c3b3148d26c2, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:50.032536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:50.034415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:50.037143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:50.043862 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 391/785] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/endpoints?service_id=1f33246da51d433c8cf3c3b3148d26c2&interface=public®ion_id=RegionOne => generated 194 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:50.051942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:50.105829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0a8d5-5afb-4790-a308-61cf4d30d752 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.170613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.171205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.171490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.171874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.182654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:50.184240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:50.199352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:50.200330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:50.201228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:50.201228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:50.201228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:50.201228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 395/786] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:50.207538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:50.229814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:50.250916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc88bb5-67e6-4eef-90a2-fce8135974c9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.313832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.314814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.315508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.316075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.317023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:50.318587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:50.325230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 392/787] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/services => generated 772 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:50.326307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:50.377899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c292801-5111-4779-bf89-9e7db07a0d7a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.399984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:50.446539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.447092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.447903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.448422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.449142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:50.451048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:50.453810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:50.468064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c4bc33-868e-4e6f-b62e-896fdcaece65 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.476638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 396/788] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:50.483114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:50.530908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.531570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.531889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.532164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.547590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:50.553979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:50.554941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:50.556095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 393/789] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:50.562076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:50.570060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:50.610655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f67ad9-4622-4428-a112-9f9068d0b021 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.659935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.660570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.660848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.661184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.671131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:50.672963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:50.689921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:50.691018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:50.691968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 397/790] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/users/swift => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:50.700383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:50.720926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db905a66-017d-4eeb-a099-fbed399d6d87 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.773527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:50.795506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.796100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.796371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.796636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.797288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:50.797586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:50.798924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:50.820085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 394/791] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:50.823863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:50.856388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a8f74-ba96-4e84-b4b3-f30372ba0c98 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:50.904507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:50.917856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:50.918467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:50.918743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:50.919036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:50.920365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:50.921466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:50.922921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:50.956748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 398/792] 10.222.0.33 () {56 vars in 1135 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:50.962784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:51.026435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a288233-e8b3-4d4b-b60d-c4ca0c3514de None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:51.098667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:51.103391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.104236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.104673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.131228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:51.132704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:51.170468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:51.172288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:51.173966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:51.173966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:51.173966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:51.173966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 395/793] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:51.182863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb376f8-1fca-4651-87e5-ab5b0ae0a34d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:51.185614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:51.254593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:51.276909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:51.277566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.277845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.278112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.314386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:51.316403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:51.356364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:51.357419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:51.358326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:51.358326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:51.358326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:51.358326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 399/794] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 395 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:51.360846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a33e52-7e90-42c7-89c7-1c8bfb583554 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:51.363486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:51.417412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:51.418877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.419234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.419525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.420517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:51.421091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:51.422869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:51.445204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 396/795] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:51.454593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:51.466510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:51.555162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a575480c-b7fd-47e4-a4a1-a94bdf686820 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:51.577607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:51.582148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c255939-e2d7-4b3b-ad7b-b5349e6154fd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:51.596801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc59a1a6-3417-4fd5-9b01-5cdfa6be3cb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:51.597408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc59a1a6-3417-4fd5-9b01-5cdfa6be3cb5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.597724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc59a1a6-3417-4fd5-9b01-5cdfa6be3cb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.597998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc59a1a6-3417-4fd5-9b01-5cdfa6be3cb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.598739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc59a1a6-3417-4fd5-9b01-5cdfa6be3cb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:51.599162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc59a1a6-3417-4fd5-9b01-5cdfa6be3cb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:51.600672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc59a1a6-3417-4fd5-9b01-5cdfa6be3cb5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:51.616326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 400/796] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:51.618445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aa91f5-15b3-4eb9-8b2a-b356f07979ab None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.618842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aa91f5-15b3-4eb9-8b2a-b356f07979ab None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.619166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aa91f5-15b3-4eb9-8b2a-b356f07979ab None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.623095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 401/797] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:51.623977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aa91f5-15b3-4eb9-8b2a-b356f07979ab None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:51.628132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25273af5-2d69-4aae-8415-356a2c9bc692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:51.628780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25273af5-2d69-4aae-8415-356a2c9bc692 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.629391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25273af5-2d69-4aae-8415-356a2c9bc692 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.629654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25273af5-2d69-4aae-8415-356a2c9bc692 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.639901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25273af5-2d69-4aae-8415-356a2c9bc692 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:51.642157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25273af5-2d69-4aae-8415-356a2c9bc692 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-25273af5-2d69-4aae-8415-356a2c9bc692 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:51.657257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:51.659547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 397/798] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:19:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d35757c-e0dd-4c43-929d-ac6c996a221d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.661832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d35757c-e0dd-4c43-929d-ac6c996a221d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.663812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d35757c-e0dd-4c43-929d-ac6c996a221d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.684457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4d35757c-e0dd-4c43-929d-ac6c996a221d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:51.686816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4d35757c-e0dd-4c43-929d-ac6c996a221d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:51.747732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:51.753042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aa91f5-15b3-4eb9-8b2a-b356f07979ab None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:51.769771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 398/799] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:51.770895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d35757c-e0dd-4c43-929d-ac6c996a221d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:51.819768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:51.820414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.820782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.821064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.841455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:51.843099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:51.861193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:51.863498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:51.865267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 402/800] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:51.868654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:51.883666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:51.908004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d35757c-e0dd-4c43-929d-ac6c996a221d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:51.977609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:51.982865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:51.983180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:51.983426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:51.984139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:51.984437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:51.985882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.025773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 399/801] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:19:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:52.035646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.076858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922a6b28-6629-43c9-82b0-6c95b6e84e29 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.144614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.145228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.145501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.146055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.146756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:52.147078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.148739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.170470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 403/802] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:52.175071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.193601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:52.210674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d841238e-1bae-4b41-92f4-0500cfd2b295 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.253011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ed3608-d87c-4298-ac9a-6bc7d72b4bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.253606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed3608-d87c-4298-ac9a-6bc7d72b4bec None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.253883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed3608-d87c-4298-ac9a-6bc7d72b4bec None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.254151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ed3608-d87c-4298-ac9a-6bc7d72b4bec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.270787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:52.296869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50023f4-5033-44c3-a457-847982603d56 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.328370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ed3608-d87c-4298-ac9a-6bc7d72b4bec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.329824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ed3608-d87c-4298-ac9a-6bc7d72b4bec None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.333514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.334142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.334416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.334684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.335297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.336648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:52.346316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:52.347449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:52.348315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 404/803] 10.222.0.33 () {56 vars in 1129 bytes} [Thu Oct 6 04:19:52 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:52.349107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.440541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:52.443069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 400/804] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:52 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 06 04:19:52.449377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4627ea-66bc-4a9f-a2e2-dfeb084595b5 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.449741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4627ea-66bc-4a9f-a2e2-dfeb084595b5 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.450123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4627ea-66bc-4a9f-a2e2-dfeb084595b5 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.451748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 401/805] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:52 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:52.456090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4627ea-66bc-4a9f-a2e2-dfeb084595b5 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.464932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71be376d-3a9e-4d0b-8c31-b7b6b98c8db3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.502689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.503366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.503639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.503906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/2e280c4b418e4aafb822a617424c7e11/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.505546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:52.506886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=2e280c4b418e4aafb822a617424c7e11, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.508731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.534997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 405/806] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:52 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/2e280c4b418e4aafb822a617424c7e11/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:19:52.534997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.593108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4627ea-66bc-4a9f-a2e2-dfeb084595b5 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.638286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d1db10-1bb4-4782-9fee-48bdc4a22406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.639437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1db10-1bb4-4782-9fee-48bdc4a22406 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.639864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1db10-1bb4-4782-9fee-48bdc4a22406 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.640256 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d1db10-1bb4-4782-9fee-48bdc4a22406 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.640956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d1db10-1bb4-4782-9fee-48bdc4a22406 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:52.641621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d1db10-1bb4-4782-9fee-48bdc4a22406 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.643452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d1db10-1bb4-4782-9fee-48bdc4a22406 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.651066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 402/807] 10.222.0.33 () {56 vars in 1124 bytes} [Thu Oct 6 04:19:52 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:52.652643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62548a-918b-460f-b888-1894efa000e7 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.652951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62548a-918b-460f-b888-1894efa000e7 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.653187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62548a-918b-460f-b888-1894efa000e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.664401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2d7318-8651-4667-8eba-d26bf04234fd None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.677942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4a62548a-918b-460f-b888-1894efa000e7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:52.679483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4a62548a-918b-460f-b888-1894efa000e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:52.704725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.705527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.705811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.706133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.719668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.721254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.723927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:52.740046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:52.741060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:52.742460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:52.742460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:52.742460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:52.746972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 406/808] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:52.747810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.774989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 403/809] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:52.774989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a62548a-918b-460f-b888-1894efa000e7 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.820111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:52.885593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a62548a-918b-460f-b888-1894efa000e7 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.915017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ffece1-b7e4-438c-9626-81417590b1c7 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:52.954607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.955257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.955534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.955867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.956439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:52.956735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.958113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.961144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:52.962026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:52.962502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:52.963574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:52.964417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:52.964936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:52.966564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:52.978854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 407/810] 10.222.0.33 () {56 vars in 1124 bytes} [Thu Oct 6 04:19:52 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:52.985211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:52.990099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 404/811] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:52.997527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:53.082325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:53.124482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:53.127641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbabb6c5-d7d3-4413-b002-45f2cb0bd199 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:53.144040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f4ff82-cfc1-4d80-a16a-722415523e04 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:53.170478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d797de-b025-4c1b-a2a3-637ba213d751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:53.171228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d797de-b025-4c1b-a2a3-637ba213d751 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.171423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d797de-b025-4c1b-a2a3-637ba213d751 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.171722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d797de-b025-4c1b-a2a3-637ba213d751 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.180133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d797de-b025-4c1b-a2a3-637ba213d751 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:53.182966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d797de-b025-4c1b-a2a3-637ba213d751 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:53.185728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:53.186331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.186605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.186872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.187570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:53.188918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-54d797de-b025-4c1b-a2a3-637ba213d751 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:53.195870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:53.196743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:53.198534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:53.205886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 405/812] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:53.205886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a82b6e-4457-4a93-aea8-02359eb55031 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.205886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a82b6e-4457-4a93-aea8-02359eb55031 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.205886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a82b6e-4457-4a93-aea8-02359eb55031 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.205886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 406/813] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:53.205886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a82b6e-4457-4a93-aea8-02359eb55031 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:53.231007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 408/814] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:19:52 2022] POST /identity/v3/endpoints => generated 346 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:19:53.231007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:53.348469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a82b6e-4457-4a93-aea8-02359eb55031 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:53.371811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d206f4-7cb8-43ef-b6a6-e68066262db6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:53.388493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6784856-6428-4c32-9efe-2118a0fc13ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:53.389171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:53.389769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6784856-6428-4c32-9efe-2118a0fc13ef None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.390130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6784856-6428-4c32-9efe-2118a0fc13ef None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.390406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6784856-6428-4c32-9efe-2118a0fc13ef None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.406782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6784856-6428-4c32-9efe-2118a0fc13ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-d6784856-6428-4c32-9efe-2118a0fc13ef None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:53.412934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:53.413900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:53.414741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:53.417878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 407/815] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:53.419116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea436b00-f335-4a31-9386-11e41c3a0411 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.419384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea436b00-f335-4a31-9386-11e41c3a0411 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.420121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea436b00-f335-4a31-9386-11e41c3a0411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.426030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:53.427348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.427348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.427348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.427952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:53.428260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:53.429937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:53.443895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ea436b00-f335-4a31-9386-11e41c3a0411 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:53.444904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 409/816] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:53.445735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:53.450411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ea436b00-f335-4a31-9386-11e41c3a0411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:53.465425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:53.540215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 408/817] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:53.540751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea436b00-f335-4a31-9386-11e41c3a0411 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:53.576666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568685e-aca6-4515-aadd-bedf4e6e2bed None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:53.648259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:53.648879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.649153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.649422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.650093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:53.653484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:53.653484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:53.671371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 410/818] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:53.679928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:53.713951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea436b00-f335-4a31-9386-11e41c3a0411 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:53.754357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:53.755378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.760804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.761293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:53.761429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.762265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=c8d2db883de44964b6e15b85a97726b5) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:53.762377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:53.765593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:53.772824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 409/819] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:53 2022] GET /identity/v3/role_assignments?user.id=c8d2db883de44964b6e15b85a97726b5&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:53.848703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:53.867387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:53.877591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cf5282-7b51-4307-8a1d-fc41b191c8f4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:53.922413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:53.923056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:53.923339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:53.924872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:53.938789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:53.940320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:53.963663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:53.964702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:53.965633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:53.965633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:53.965633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:53.965633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 411/820] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:53.970683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:54.100035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:54.118895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8414b355-4e83-405e-a4a8-f6dde61ac9ff None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.164939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b00f13-522f-4f61-8927-72eef4f34838 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.165164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:54.165775 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:54.166907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:54.167238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:54.167858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:54.168160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:54.170843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:54.188130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 412/821] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:54.195821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:54.206144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:54.206815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:54.207120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:54.207391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] PATH_INFO: `/v3/services/network` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:54.207971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:54.209458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:54.226524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:54.227639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:54.229026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 410/822] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:53 2022] GET /identity/v3/services/network => generated 88 bytes in 380 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:54.237208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:54.361288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba91c147-986d-4f5b-ad3e-77ce0e54d38d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.371320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:54.398929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbe8232-805a-4e62-8e46-cd678c259161 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.425045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:54.425752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:54.426225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:54.426334 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:54.437291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:54.438278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:54.438767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:54.439252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:54.439454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:54.440256 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] RBAC: Adding query filter params (name=network) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:54.440711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:54.440938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:54.442472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:54.448166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 411/823] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:19:54 2022] GET /identity/v3/services?name=network => generated 124 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:54.453752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:54.469336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:54.470361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:54.471329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 413/824] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:19:54 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:54.478711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:54.491805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:54.625367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1f7d5b-87f5-49d3-baa1-9bb9b1b0ff64 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.650402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d5bc62-c4e8-43b9-901e-e94d4fccae0d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.688027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:54.688758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:54.688971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:54.689240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:54.689961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:54.690263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:54.691739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:54.711603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 414/825] 10.222.0.33 () {56 vars in 1149 bytes} [Thu Oct 6 04:19:54 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:54.715700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:54.716304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:54.716576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:54.716842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:54.717502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] RBAC: Adding query filter params (type=network) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:54.717763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:54.719124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:54.721146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:54.735826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 412/826] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:19:54 2022] GET /identity/v3/services?type=network => generated 350 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:54.741906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:54.745201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:54.846845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:54.917023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31cccea-4348-4cda-a94b-bc2930305020 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.961939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:54.962490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:54.962765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:54.963143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:54.967408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac444786-643a-4ffe-8d91-06b6a5f4cf57 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:54.976627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:54.978101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:54.998284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:54.999335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:55.000235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:55.000235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:55.000235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:55.000235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 415/827] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:55.006378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:55.034261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f4f956-f0c3-4b5a-b376-d2f4f869cf42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:55.035318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f4f956-f0c3-4b5a-b376-d2f4f869cf42 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.035857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f4f956-f0c3-4b5a-b376-d2f4f869cf42 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.037075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f4f956-f0c3-4b5a-b376-d2f4f869cf42 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.037962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f4f956-f0c3-4b5a-b376-d2f4f869cf42 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:55.039708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f4f956-f0c3-4b5a-b376-d2f4f869cf42 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:55.078308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 413/828] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:19:54 2022] POST /identity/v3/endpoints => generated 337 bytes in 337 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:19:55.084354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62df705-f94e-4e53-a542-9d81b38f761b None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.084962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62df705-f94e-4e53-a542-9d81b38f761b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.085465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62df705-f94e-4e53-a542-9d81b38f761b None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.087376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 414/829] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:55 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:55.093422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a058a12-9b92-4bc2-aa9f-2f119f8595b7 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.094017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a058a12-9b92-4bc2-aa9f-2f119f8595b7 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.094583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a058a12-9b92-4bc2-aa9f-2f119f8595b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.134361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4a058a12-9b92-4bc2-aa9f-2f119f8595b7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:55.136121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4a058a12-9b92-4bc2-aa9f-2f119f8595b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:55.164351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3464977-5ed0-4166-a400-c08c0927b419 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:55.188173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:55.219750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:55.220625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.221079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.221475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.222265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:55.222719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:55.224316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:55.240614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 416/830] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:55.244643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:55.253060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:55.277318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 415/831] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:55 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:55.414574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b153b75-54ed-49d0-a79f-30488c924ef4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:55.459610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:55.460631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.461116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.461682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.469169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:55.470558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:55.482356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:55.483373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:55.484276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 417/832] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:55.489944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a058a12-9b92-4bc2-aa9f-2f119f8595b7 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:55.597602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a058a12-9b92-4bc2-aa9f-2f119f8595b7 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:55.605105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:55.635177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:55.636071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.636252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.636502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.637201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:55.637501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:55.638945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:55.643976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:55.659524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 416/833] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:55.662333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:55.717998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:55.726300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad5947-8219-4db6-b2d0-29fdbebaf6d5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:55.764498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:55.765125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.765405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.765669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.776563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:55.777549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d2f51-7857-4664-8e49-bde8c71a6597 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:55.782727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:55.784940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:55.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 418/834] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:55.790251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:55.815741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3437276-ee12-4339-8082-5f50405a63d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:55.816518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3437276-ee12-4339-8082-5f50405a63d9 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.816800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3437276-ee12-4339-8082-5f50405a63d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.817138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3437276-ee12-4339-8082-5f50405a63d9 None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/371190691e284514bafcba1bdd199fc6/roles/bcc8e52e7c184e5984637371b5614639` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.835681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3437276-ee12-4339-8082-5f50405a63d9 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=371190691e284514bafcba1bdd199fc6, role_id=bcc8e52e7c184e5984637371b5614639)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:55.837201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3437276-ee12-4339-8082-5f50405a63d9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:55.856195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 417/835] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:19:55 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/371190691e284514bafcba1bdd199fc6/roles/bcc8e52e7c184e5984637371b5614639 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:19:55.857492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73386ab-927a-4521-850a-d5596a0ef39a None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.857849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73386ab-927a-4521-850a-d5596a0ef39a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.858126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73386ab-927a-4521-850a-d5596a0ef39a None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.859815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 418/836] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:55.870649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b9cd99-f428-40bf-9899-199aa501d02f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:55.871060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b9cd99-f428-40bf-9899-199aa501d02f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:55.871288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b9cd99-f428-40bf-9899-199aa501d02f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:55.899075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-41b9cd99-f428-40bf-9899-199aa501d02f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:55.900473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-41b9cd99-f428-40bf-9899-199aa501d02f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:55.995019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 419/837] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:55 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:19:56.034479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:56.044621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169f34bf-d0d7-4b51-9ef3-3df177625137 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.084934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.085540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.085812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.086078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.086736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:56.087619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.088600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.098885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 419/838] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:19:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:56.106523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b9cd99-f428-40bf-9899-199aa501d02f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.141964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:56.219014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b9cd99-f428-40bf-9899-199aa501d02f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.251904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.252445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.252716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.252982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.263308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.264731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:56.279840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:56.280864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:56.281991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:56.281991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:56.281991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:56.281991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 420/839] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:56.288675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.315889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.410918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f620f3-9f41-44a1-8103-c0b1d8563902 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.419123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:56.448226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1cfa66-4d18-4010-90be-2708cd5ef927 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.459400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.460064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.460295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.460564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.461219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:56.461516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.463576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.479076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 420/840] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:56.484593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.499559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.500684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.501175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.502081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.503162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.505157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.511350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:56.518002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:56.519076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:19:56.520043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:56.520825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 421/841] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:56.527810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.613444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912b5642-be59-42d6-bf22-b6b3285dd0e5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.648201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cbcce6-3e91-4919-888f-b85a6c310cf7 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.650015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7946f39c-b2a4-4986-9636-24d0f5f33dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.650737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7946f39c-b2a4-4986-9636-24d0f5f33dd6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.651040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7946f39c-b2a4-4986-9636-24d0f5f33dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.651312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7946f39c-b2a4-4986-9636-24d0f5f33dd6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.658985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7946f39c-b2a4-4986-9636-24d0f5f33dd6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.660432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7946f39c-b2a4-4986-9636-24d0f5f33dd6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-7946f39c-b2a4-4986-9636-24d0f5f33dd6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:56.672642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:56.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 421/842] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/users/swift => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:56.675296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e8c5da-69d8-486c-badb-6fde8e2aa0fa None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.675959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e8c5da-69d8-486c-badb-6fde8e2aa0fa None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.675959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e8c5da-69d8-486c-badb-6fde8e2aa0fa None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.677101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 422/843] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:56.682231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e8c5da-69d8-486c-badb-6fde8e2aa0fa None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.692040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f2d9cd-5a3c-4742-8f12-941bb73dd438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.692642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f2d9cd-5a3c-4742-8f12-941bb73dd438 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.692918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f2d9cd-5a3c-4742-8f12-941bb73dd438 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.693581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f2d9cd-5a3c-4742-8f12-941bb73dd438 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.694220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f2d9cd-5a3c-4742-8f12-941bb73dd438 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:56.694840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f2d9cd-5a3c-4742-8f12-941bb73dd438 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.696229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f2d9cd-5a3c-4742-8f12-941bb73dd438 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.706077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 422/844] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:56.706077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03731f07-309f-4c18-af74-8007b0d408d2 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.706077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03731f07-309f-4c18-af74-8007b0d408d2 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.706077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03731f07-309f-4c18-af74-8007b0d408d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.721817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-03731f07-309f-4c18-af74-8007b0d408d2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:56.723044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-03731f07-309f-4c18-af74-8007b0d408d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:56.725147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:56.770886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:56.799005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 423/845] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:56 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:56.799005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03731f07-309f-4c18-af74-8007b0d408d2 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.802348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e8c5da-69d8-486c-badb-6fde8e2aa0fa None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.843350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.843957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.844234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.844505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.845229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:56.845533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.846985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.874889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 423/846] 10.222.0.33 () {56 vars in 1135 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:56.880530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:56.926928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03731f07-309f-4c18-af74-8007b0d408d2 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:56.975105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:56.976494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:56.976948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:56.977416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:56.979258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:56.981037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:56.982390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:56.991099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 424/847] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:57.013371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051a1593-9a4c-49e7-924e-6fdaf5c66d74 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:57.052243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:57.053227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:57.053719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:57.054190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:57.058135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:57.091199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:57.094419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:57.144575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:57.146501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:57.223732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5137b472-ce1b-4742-92e6-2b4a63461142 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:57.266637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:57.267321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:57.267597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:57.267925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:57.268506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:57.270065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:57.274707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 424/848] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:56 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:57.283168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:57.284146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:19:57.285048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:57.285807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:57.285879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 425/849] 10.222.0.33 () {56 vars in 1132 bytes} [Thu Oct 6 04:19:57 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:57.287762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:57.328099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:57.359396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:57.406675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd13f0e-9b7c-4a2d-9668-3454e9abaec5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:57.411909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed89396-e70e-483f-921c-7f06c0d9cbc5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:57.448734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:57.449519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:57.449863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:57.450231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:57.450948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:57.451363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:57.453145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:57.455570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:57.456373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:57.457029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:57.457442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:57.459113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 426/850] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:19:57 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:57.464704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:57.469355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:57.471390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:57.487496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:57.488471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:57.489371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:57.489371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:57.489371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:19:57.491503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 425/851] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:57.497619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:57.595165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:57.610031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765444f6-8435-4bed-828b-5be87c251caa None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:57.617977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23af9b8e-9bab-4c22-96d1-88b0ae55980c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:57.642316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:57.649302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:57.650690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:57.651824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:57.652202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:57.653094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:57.653447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:57.655150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:57.660379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 427/852] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:19:57 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:57.671067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:57.671067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:57.671067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:57.671067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:57.671511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:57.671511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:57.671511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:57.686603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 426/853] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:57.691846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:57.813329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66419f77-ec49-46ee-8339-e2a76814fd5d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:57.852551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:57.853094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:57.853453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:57.853725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:57.860513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:57.862183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:19:57.873416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:57.874338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:57.875326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 428/854] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:19:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:19:57.882133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:57.990764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-005a1d06-a07b-4fbe-bc24-b6185eff066a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:58.027618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07975fff-4f6b-48a7-ae3b-93ceef799521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:58.028204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07975fff-4f6b-48a7-ae3b-93ceef799521 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:58.028473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07975fff-4f6b-48a7-ae3b-93ceef799521 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:58.028736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07975fff-4f6b-48a7-ae3b-93ceef799521 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:58.029416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07975fff-4f6b-48a7-ae3b-93ceef799521 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:58.029710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07975fff-4f6b-48a7-ae3b-93ceef799521 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:58.031167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07975fff-4f6b-48a7-ae3b-93ceef799521 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:58.043717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 427/855] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:19:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:19:58.049730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:58.115402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:58.162191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9778006d-7cf9-4766-a023-10bcbb2be14f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:58.187777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853ec9c6-97b3-41f8-914d-f5c5e8af7ab5 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:58.188039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853ec9c6-97b3-41f8-914d-f5c5e8af7ab5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:58.188842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853ec9c6-97b3-41f8-914d-f5c5e8af7ab5 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:58.189921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 428/856] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:58.197814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443ef11-88fb-41ee-b038-dee2d10640e9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:58.198309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443ef11-88fb-41ee-b038-dee2d10640e9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:58.198690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443ef11-88fb-41ee-b038-dee2d10640e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:58.202767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:58.203650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:58.204606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:58.205413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:58.207510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=2e280c4b418e4aafb822a617424c7e11) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:58.207837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:58.209412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:58.216918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:58.222983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 429/857] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:19:58 2022] GET /identity/v3/role_assignments?user.id=2e280c4b418e4aafb822a617424c7e11&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:58.223634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b443ef11-88fb-41ee-b038-dee2d10640e9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:58.226629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b443ef11-88fb-41ee-b038-dee2d10640e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:58.308769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 429/858] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:19:58.576697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:58.679641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e48c522-5ec8-4037-9083-a90e1af885d4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:58.696731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:58.714864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:58.715515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:58.715829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:58.716099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:58.726332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:19:58.732068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:58.733015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:58.734290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 430/859] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:19:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:58.739715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443ef11-88fb-41ee-b038-dee2d10640e9 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:58.841712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443ef11-88fb-41ee-b038-dee2d10640e9 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:58.880784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:58.881400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:58.881678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:58.882026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:58.882931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:58.883288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:58.884772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:58.887766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:58.894525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 430/860] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:19:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:58.900984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:59.006176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475d761a-38b7-47e6-a8b7-44d6a787ee0f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:59.041562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:59.042192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.042472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.042750 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.055463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:59.057730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:19:59.072787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:59.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:19:59.075433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:19:59.075433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:19:59.075433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:59.075433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 431/861] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:59.080066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:59.188139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1400fdc2-dc78-46de-a8bb-dfd7588c9cab None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:59.224581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ace649-389d-4fb0-b9e4-740e55453d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:59.225394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ace649-389d-4fb0-b9e4-740e55453d15 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.225813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ace649-389d-4fb0-b9e4-740e55453d15 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.226239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ace649-389d-4fb0-b9e4-740e55453d15 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.227061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ace649-389d-4fb0-b9e4-740e55453d15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:59.227523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ace649-389d-4fb0-b9e4-740e55453d15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:59.229297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ace649-389d-4fb0-b9e4-740e55453d15 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:59.242201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 431/862] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:19:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:59.252032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:59.285959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:59.361252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09b6a0-7755-4f09-93d0-f3e489a655d8 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:59.398809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:59.399422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28457959-34e5-4c70-9a28-560091733737 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.399817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.400140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.400899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.401223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28457959-34e5-4c70-9a28-560091733737 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.401707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28457959-34e5-4c70-9a28-560091733737 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.407199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 432/863] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:19:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:19:59.408786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:59.409162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8de5268-20fd-4af6-9b76-57c17710073e None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.409662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8de5268-20fd-4af6-9b76-57c17710073e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.410154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8de5268-20fd-4af6-9b76-57c17710073e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.410269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:19:59.423399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:19:59.424434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:19:59.425412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:19:59.428079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 432/864] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:19:59 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:19:59.429621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a8de5268-20fd-4af6-9b76-57c17710073e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:19:59.430823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a8de5268-20fd-4af6-9b76-57c17710073e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:19:59.443057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:59.499245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:59.517672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 433/865] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:19:59 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:19:59.538997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:59.549283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dd8ff3-f931-41cb-a825-d4551a5ba9ea None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:59.584817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:59.585697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.586146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.586560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.587468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:19:59.588030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:59.589794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:59.609249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 433/866] 10.222.0.33 () {56 vars in 1149 bytes} [Thu Oct 6 04:19:59 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:19:59.615009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8de5268-20fd-4af6-9b76-57c17710073e None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:59.720484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8de5268-20fd-4af6-9b76-57c17710073e None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:59.755625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:59.756235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.756504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.756781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.782377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:59.820787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:19:59.840588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:59.842037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:59.913264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3f5d2-24d0-4a0b-8ee2-23cf2a6ca739 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:19:59.952684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b2c1a4-3acc-4e42-85c4-15140f1cb29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:19:59.953348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b2c1a4-3acc-4e42-85c4-15140f1cb29e None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:19:59.953628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b2c1a4-3acc-4e42-85c4-15140f1cb29e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:19:59.953898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b2c1a4-3acc-4e42-85c4-15140f1cb29e None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:19:59.954566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b2c1a4-3acc-4e42-85c4-15140f1cb29e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:19:59.956107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b2c1a4-3acc-4e42-85c4-15140f1cb29e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:19:59.966017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 434/867] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:19:59 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:19:59.971569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:19:59.975059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 434/868] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:19:59 2022] POST /identity/v3/services => generated 251 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:00.073233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386a69a-2d26-4017-aeb7-c1761a289917 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.073602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386a69a-2d26-4017-aeb7-c1761a289917 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.073784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386a69a-2d26-4017-aeb7-c1761a289917 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.075465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 435/869] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:00.081694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d41486c-10d7-47f9-a486-39234372d202 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.082332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d41486c-10d7-47f9-a486-39234372d202 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.082815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d41486c-10d7-47f9-a486-39234372d202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.087109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:00.101522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b9781-5525-42fc-9722-9e2757e7df0b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:00.105662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8d41486c-10d7-47f9-a486-39234372d202 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:00.107207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8d41486c-10d7-47f9-a486-39234372d202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:00.110779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:00.151458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:00.152104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.152375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.152639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.164879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:00.166487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:00.183447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:00.184494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:00.185409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:00.185409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:00.185409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:00.189746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 435/870] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:19:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:00.190803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:00.199057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 436/871] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:00 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:00.302334 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ae14a1-514a-40ec-8d0e-26a1bfa43d20 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:00.337302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:00.338035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.338310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.338581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.339247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:00.339546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:00.340916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:00.343550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:00.354886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 436/872] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:00.360465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d41486c-10d7-47f9-a486-39234372d202 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:00.468564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:00.472188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d41486c-10d7-47f9-a486-39234372d202 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:00.517934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6843475a-5fc9-48bf-b8ab-42aedaf3caff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:00.518883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6843475a-5fc9-48bf-b8ab-42aedaf3caff None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.519329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6843475a-5fc9-48bf-b8ab-42aedaf3caff None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.519765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6843475a-5fc9-48bf-b8ab-42aedaf3caff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.529104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6843475a-5fc9-48bf-b8ab-42aedaf3caff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:00.530868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6843475a-5fc9-48bf-b8ab-42aedaf3caff None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:00.532878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-6843475a-5fc9-48bf-b8ab-42aedaf3caff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:00.544202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:00.545328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:00.546410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:00.548132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 437/873] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:00.549441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8352040-91e5-40e5-bec5-aca6089cb81e None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.549714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8352040-91e5-40e5-bec5-aca6089cb81e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.549980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8352040-91e5-40e5-bec5-aca6089cb81e None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.551396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 438/874] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:00.555882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8352040-91e5-40e5-bec5-aca6089cb81e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:00.597563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d31be7-080d-4f2d-b9a6-25e8d337700e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:00.634092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1446cd92-e07f-49f9-8ff4-9501d4a7de84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:00.635121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1446cd92-e07f-49f9-8ff4-9501d4a7de84 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.635121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1446cd92-e07f-49f9-8ff4-9501d4a7de84 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.635418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1446cd92-e07f-49f9-8ff4-9501d4a7de84 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.635944 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1446cd92-e07f-49f9-8ff4-9501d4a7de84 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:00.637311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1446cd92-e07f-49f9-8ff4-9501d4a7de84 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:00.655038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 437/875] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:20:00 2022] POST /identity/v3/services => generated 269 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:00.669328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706fa8f6-8641-45c3-881c-ff9bccd0f39d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.669718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706fa8f6-8641-45c3-881c-ff9bccd0f39d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.669946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706fa8f6-8641-45c3-881c-ff9bccd0f39d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.684712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8352040-91e5-40e5-bec5-aca6089cb81e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:00.695088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-706fa8f6-8641-45c3-881c-ff9bccd0f39d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:00.696380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-706fa8f6-8641-45c3-881c-ff9bccd0f39d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:00.733052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:00.734923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.735471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.736205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.737499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:00.738018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:00.740091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:00.756797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 439/876] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:20:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:00.765508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:00.771194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:00.795251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 438/877] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:00 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:00.831429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:00.877458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcf7ebc-a0fc-4aa7-8ade-3a0511b8d39d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:00.916391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:00.917213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:00.917453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:00.917726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:00.918447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=371190691e284514bafcba1bdd199fc6) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:00.918749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:00.920213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:00.926583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 440/878] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:00 2022] GET /identity/v3/role_assignments?user.id=371190691e284514bafcba1bdd199fc6&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:01.091283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706fa8f6-8641-45c3-881c-ff9bccd0f39d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:01.205504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706fa8f6-8641-45c3-881c-ff9bccd0f39d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:01.248591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:01.249236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:01.249511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:01.249783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:01.250370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:01.251777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:01.261016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:01.262083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:01.262938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 439/879] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:01 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:01.268282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:01.377747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67d219f-766b-41ba-a5d5-2bfa719028ad None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:01.396065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:01.415437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfc412-29c5-4b9a-bf77-a372af06eca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:01.416638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfc412-29c5-4b9a-bf77-a372af06eca2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:01.417154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfc412-29c5-4b9a-bf77-a372af06eca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:01.417619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfc412-29c5-4b9a-bf77-a372af06eca2 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:01.418437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfc412-29c5-4b9a-bf77-a372af06eca2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:01.418936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfc412-29c5-4b9a-bf77-a372af06eca2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:01.420529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfc412-29c5-4b9a-bf77-a372af06eca2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:01.425997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 441/880] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:01 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:01.431916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:01.485476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:01.537340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea5912-a468-4368-9333-6f7bbf32678b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:01.572538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4a5923-4a01-4a20-afa9-50b5abe8e343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:01.573139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4a5923-4a01-4a20-afa9-50b5abe8e343 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:01.573413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4a5923-4a01-4a20-afa9-50b5abe8e343 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:01.573697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4a5923-4a01-4a20-afa9-50b5abe8e343 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:01.574354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4a5923-4a01-4a20-afa9-50b5abe8e343 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:01.574657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4a5923-4a01-4a20-afa9-50b5abe8e343 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:01.576113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4a5923-4a01-4a20-afa9-50b5abe8e343 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:01.580927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 440/881] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:01 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:02.317540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a152f-16ae-4605-ab03-4924d2ca127d None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:02.318090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a152f-16ae-4605-ab03-4924d2ca127d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:02.318486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a152f-16ae-4605-ab03-4924d2ca127d None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:02.320221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 442/882] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:02.325565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b12d5a-f4ee-4db8-ae8a-8edfa2ce30bf None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:02.326010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b12d5a-f4ee-4db8-ae8a-8edfa2ce30bf None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:02.326299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b12d5a-f4ee-4db8-ae8a-8edfa2ce30bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:02.345472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c0b12d5a-f4ee-4db8-ae8a-8edfa2ce30bf None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:02.346647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c0b12d5a-f4ee-4db8-ae8a-8edfa2ce30bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:02.414717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 441/883] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:02 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:02.684581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a152f-16ae-4605-ab03-4924d2ca127d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:02.789771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a152f-16ae-4605-ab03-4924d2ca127d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:02.824842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a705c0-7652-4379-b257-5cb746c040ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:02.825469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a705c0-7652-4379-b257-5cb746c040ad None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:02.825741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a705c0-7652-4379-b257-5cb746c040ad None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:02.826010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a705c0-7652-4379-b257-5cb746c040ad None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:02.826646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a705c0-7652-4379-b257-5cb746c040ad None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:02.828034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a705c0-7652-4379-b257-5cb746c040ad None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-83a705c0-7652-4379-b257-5cb746c040ad None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:02.837245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:02.838830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:02.839714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 443/884] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:20:02 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:02.845230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b12d5a-f4ee-4db8-ae8a-8edfa2ce30bf None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:02.896499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:02.960117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b12d5a-f4ee-4db8-ae8a-8edfa2ce30bf None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:03.005139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:03.005749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.006024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.006297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.006928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:03.007258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:03.008684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:03.011784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e034ab-52b3-4ad9-be4a-1831f952c854 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.012341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e034ab-52b3-4ad9-be4a-1831f952c854 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.012796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e034ab-52b3-4ad9-be4a-1831f952c854 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.014383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 442/885] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:20:02 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:03.018460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 444/886] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:03.019860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:03.020339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f510fd31-9c58-41bd-ab2e-9d9cf466ffa8 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.020933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f510fd31-9c58-41bd-ab2e-9d9cf466ffa8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.021413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f510fd31-9c58-41bd-ab2e-9d9cf466ffa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.044052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f510fd31-9c58-41bd-ab2e-9d9cf466ffa8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:03.046133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f510fd31-9c58-41bd-ab2e-9d9cf466ffa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:03.094501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:03.122352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 445/887] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:03 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:03.136399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251ee999-acd9-4d44-94f6-5f10e41b1d94 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:03.146084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:03.174073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:03.175033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.175477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.175926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.176687 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:03.177170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:03.178670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:03.184191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 443/888] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:20:03 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:03.190136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f510fd31-9c58-41bd-ab2e-9d9cf466ffa8 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:03.299486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f510fd31-9c58-41bd-ab2e-9d9cf466ffa8 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:03.334928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeec6cf-50f4-414a-b96c-968d3d9218c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:03.335610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeec6cf-50f4-414a-b96c-968d3d9218c1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.335851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeec6cf-50f4-414a-b96c-968d3d9218c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.336079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeec6cf-50f4-414a-b96c-968d3d9218c1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.336834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeec6cf-50f4-414a-b96c-968d3d9218c1 None admin] RBAC: Adding query filter params (interface=public, service_id=1a05b299f8f44a6eb7064ffbbb5b75e8, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:03.337133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeec6cf-50f4-414a-b96c-968d3d9218c1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:03.339049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeec6cf-50f4-414a-b96c-968d3d9218c1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:03.344748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 446/889] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:20:03 2022] GET /identity/v3/endpoints?service_id=1a05b299f8f44a6eb7064ffbbb5b75e8&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:03.350519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:03.367594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f35d1c-b8d4-479a-bf7f-15272cf9dfe5 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.368160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f35d1c-b8d4-479a-bf7f-15272cf9dfe5 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.368597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f35d1c-b8d4-479a-bf7f-15272cf9dfe5 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.370429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 447/890] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:03.375746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1b4dd8-64e9-4754-a700-ab87c29de83d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.376278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1b4dd8-64e9-4754-a700-ab87c29de83d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.376770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1b4dd8-64e9-4754-a700-ab87c29de83d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.395729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-df1b4dd8-64e9-4754-a700-ab87c29de83d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:03.397279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-df1b4dd8-64e9-4754-a700-ab87c29de83d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:03.434206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:03.464804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec88456-2b1f-4754-96cb-bbca8583411d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:03.469270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 448/891] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:03 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:03.475251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1b4dd8-64e9-4754-a700-ab87c29de83d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:03.502317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:03.503009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.503301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.503565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.504163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:03.505524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:03.519452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 444/892] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:20:03 2022] GET /identity/v3/services => generated 1268 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:20:03.596891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1b4dd8-64e9-4754-a700-ab87c29de83d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:03.639145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:03.639791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.640072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.640346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.640945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:03.642359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:03.653644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:03.654709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:20:03.655682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:03.659127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 449/893] 10.222.0.33 () {56 vars in 1132 bytes} [Thu Oct 6 04:20:03 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:03.661849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:03.707865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:03.747717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:03.785485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36158c3-e9ed-4a94-a36a-627190fa9a7c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:03.788573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:03.824114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85e22a0-0ee4-4be5-97bb-60fc103536a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:03.824726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85e22a0-0ee4-4be5-97bb-60fc103536a7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.825026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85e22a0-0ee4-4be5-97bb-60fc103536a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.825268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85e22a0-0ee4-4be5-97bb-60fc103536a7 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.826323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85e22a0-0ee4-4be5-97bb-60fc103536a7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:03.826323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85e22a0-0ee4-4be5-97bb-60fc103536a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:03.827575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85e22a0-0ee4-4be5-97bb-60fc103536a7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:03.832508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 445/894] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:03 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:03.836671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1988ded2-b2c8-49ab-9e13-3a2ce552c1bf None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.837200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1988ded2-b2c8-49ab-9e13-3a2ce552c1bf None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.837547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1988ded2-b2c8-49ab-9e13-3a2ce552c1bf None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.839425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 446/895] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:03.843724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1988ded2-b2c8-49ab-9e13-3a2ce552c1bf None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:03.872320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22815d9a-31c2-4921-a33a-7f137b3f6226 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:03.909879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b63f68-cd4e-405d-a4f2-cb1392eab0d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:03.910503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b63f68-cd4e-405d-a4f2-cb1392eab0d9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.910778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b63f68-cd4e-405d-a4f2-cb1392eab0d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.911079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b63f68-cd4e-405d-a4f2-cb1392eab0d9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.922156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b63f68-cd4e-405d-a4f2-cb1392eab0d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-f4b63f68-cd4e-405d-a4f2-cb1392eab0d9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:03.927349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:03.928308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 450/896] 10.222.0.33 () {56 vars in 1120 bytes} [Thu Oct 6 04:20:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:03.929905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b0d38f-299e-4ea1-bfdd-d69933784a45 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:03.930702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b0d38f-299e-4ea1-bfdd-d69933784a45 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:03.930702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b0d38f-299e-4ea1-bfdd-d69933784a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:03.949013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d8b0d38f-299e-4ea1-bfdd-d69933784a45 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:03.950204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d8b0d38f-299e-4ea1-bfdd-d69933784a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:03.964537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:03.968978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1988ded2-b2c8-49ab-9e13-3a2ce552c1bf None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.004680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.005261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.005629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.005815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.006447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:04.006751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.008155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.015403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 447/897] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:03 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:04.016262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.039027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 451/898] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:03 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:04.039027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b0d38f-299e-4ea1-bfdd-d69933784a45 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.070627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:04.138306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe647d1-bc56-479e-b63a-e48353ee0d60 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.161911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b0d38f-299e-4ea1-bfdd-d69933784a45 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.178739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.179620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.179904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.180178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.181842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:04.182148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.183790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.194704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 448/899] 10.222.0.33 () {56 vars in 1115 bytes} [Thu Oct 6 04:20:04 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:04.200551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.203854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.205596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.206027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.206460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.207358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] RBAC: Adding query filter params (interface=public, service_id=58679f68bb1644ecb1234404613c07fb, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:04.207822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.209331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.219041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 452/900] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:20:04 2022] GET /identity/v3/endpoints?service_id=58679f68bb1644ecb1234404613c07fb&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:04.223475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.245964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:04.329233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:04.342391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c160c3-b248-4af2-9e94-f51ecdef7af9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.350851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ffa688-eb1a-4eb7-923f-fc1f5ddf9386 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.377758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.378352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.378628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.378912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.379536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.380937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.386546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 453/901] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:20:04 2022] GET /identity/v3/services => generated 1268 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:20:04.391647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.392689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.393173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.393622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.396036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.411641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.414267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:04.438830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:04.439876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:04.440780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:04.440780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:04.440780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:04.443505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 449/902] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:04.449365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.520676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0428145a-e7cf-4b6d-83ee-f517bb6be6f1 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.547902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:04.560314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.560924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.561201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.561472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.562122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.563515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.578984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6c46a8-88eb-435d-a96d-481c12a2e54d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.583873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 454/903] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:20:04 2022] POST /identity/v3/services => generated 241 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:04.630844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.631559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.631837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.632107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.632755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:04.633115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.636083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.650111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 450/904] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:04.655604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.674695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:04.779994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c234314-6ec9-4e15-8d92-67a2cc496cd6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:04.818921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:04.819663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:04.819944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:04.820212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:04.828892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:04.830317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:20:04.842350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:04.843380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:04.844324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 455/905] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:20:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:04.849051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:04.962329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc2336-3e4f-43d9-ab04-5fe9c56e2019 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:05.002072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:05.002685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:05.002982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:05.003262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:05.003991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:05.004296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:05.005726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:05.007386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:05.026989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 451/906] 10.222.0.33 () {56 vars in 1149 bytes} [Thu Oct 6 04:20:04 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:05.032353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:05.107138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:05.139338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d237e622-8e1c-496f-840f-3265f29343d5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:05.174345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:05.175342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:05.175835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:05.176321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:05.247186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:05.248616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:05.318343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:05.351060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 456/907] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:05 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:20:05.357330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:05.474166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e7242-87ab-4206-a96a-b1ed9615c4f3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:05.511223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:05.511857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:05.512134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:05.512404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:05.522910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:05.524394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:05.540610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:05.541594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:05.542554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:05.542554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:05.542554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:05.542554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 452/908] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:05.548839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:05.654595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dbc3e2-c722-47d7-8d34-1588c7bc9128 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:05.692634 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdd3edc-a562-47fa-b078-ec533b125a68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:05.693641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdd3edc-a562-47fa-b078-ec533b125a68 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:05.694118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdd3edc-a562-47fa-b078-ec533b125a68 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:05.694688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdd3edc-a562-47fa-b078-ec533b125a68 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:05.695710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdd3edc-a562-47fa-b078-ec533b125a68 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:05.696252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdd3edc-a562-47fa-b078-ec533b125a68 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:05.698767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdd3edc-a562-47fa-b078-ec533b125a68 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:05.712743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 457/909] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:05.718567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:05.768285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:05.803593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e67f67a-0351-41a9-a41f-770656e7ad94 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:05.803939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e67f67a-0351-41a9-a41f-770656e7ad94 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:05.804175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e67f67a-0351-41a9-a41f-770656e7ad94 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:05.807085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 458/910] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:05.811616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8298a-5964-4d48-8bdd-b6502ed3dcda None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:05.811899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8298a-5964-4d48-8bdd-b6502ed3dcda None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:05.812173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8298a-5964-4d48-8bdd-b6502ed3dcda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:05.831284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2ee8298a-5964-4d48-8bdd-b6502ed3dcda None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:05.832661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2ee8298a-5964-4d48-8bdd-b6502ed3dcda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:05.835527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0804af-4f91-452c-b1e8-76f5927fff88 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:05.863072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:05.873900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:05.874467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:05.874697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:05.874986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:05.883125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:05.885289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:05.897715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:05.898621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:05.900691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 453/911] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:05.907975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:05.918184 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 459/912] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:06.016606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1606b3e-5a50-4b00-a459-df3dc171f846 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:06.018941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:06.059142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:06.059752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.060031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.060301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.061052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:06.061359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:06.062898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:06.077176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 454/913] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:20:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:06.082812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee8298a-5964-4d48-8bdd-b6502ed3dcda None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:06.186633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee8298a-5964-4d48-8bdd-b6502ed3dcda None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:06.192843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:06.230641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:06.231548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.231833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.232104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.232829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] RBAC: Adding query filter params (role.id=f1d374a285264fe38d7173e7d7603f55, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=371190691e284514bafcba1bdd199fc6) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:06.233132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:06.234546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:06.245615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 460/914] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:06 2022] GET /identity/v3/role_assignments?user.id=371190691e284514bafcba1bdd199fc6&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=f1d374a285264fe38d7173e7d7603f55 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:06.320533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829dfcc-e5f9-4dfd-be56-188ceeed783e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:06.363298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:06.364106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.364282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.364555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.365161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:06.366559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:06.377105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:06.378203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:06.379267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 455/915] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:20:06 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:06.387749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:06.419417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:06.497891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a584b2-8bd6-4acb-ac89-c30ae4364b5f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:06.537982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:06.538592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.538868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.539174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.539805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:06.540104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:06.541546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:06.547138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 461/916] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:20:06 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:06.552715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:06.588793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:06.669047 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0994ef05-6ac1-475c-9c0a-3f7180213722 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:06.703608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a4c395-4973-4f67-a4b7-a04f9994623f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:06.704222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a4c395-4973-4f67-a4b7-a04f9994623f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.704513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a4c395-4973-4f67-a4b7-a04f9994623f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.704822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a4c395-4973-4f67-a4b7-a04f9994623f None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.705450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a4c395-4973-4f67-a4b7-a04f9994623f None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:06.705747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a4c395-4973-4f67-a4b7-a04f9994623f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:06.707141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a4c395-4973-4f67-a4b7-a04f9994623f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:06.713041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 456/917] 10.222.0.33 () {56 vars in 1114 bytes} [Thu Oct 6 04:20:06 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:06.720095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:06.760983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc06588d-3366-471c-890a-092dd11ddbb1 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.761154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc06588d-3366-471c-890a-092dd11ddbb1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.761431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc06588d-3366-471c-890a-092dd11ddbb1 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.766792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 457/918] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:06.768143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca72089-fec3-490d-9533-a8d3f6a7aeb3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.768567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca72089-fec3-490d-9533-a8d3f6a7aeb3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.769004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca72089-fec3-490d-9533-a8d3f6a7aeb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.788081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-2ca72089-fec3-490d-9533-a8d3f6a7aeb3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:06.789575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2ca72089-fec3-490d-9533-a8d3f6a7aeb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:06.820655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:06.832427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300b8cd-8165-4068-bb5c-ce103e08341b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:06.872251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b377a5f-7b10-41d4-87d2-ac95fb5f6354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:06.872941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b377a5f-7b10-41d4-87d2-ac95fb5f6354 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:06.873227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b377a5f-7b10-41d4-87d2-ac95fb5f6354 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:06.873504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b377a5f-7b10-41d4-87d2-ac95fb5f6354 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:06.874172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b377a5f-7b10-41d4-87d2-ac95fb5f6354 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:06.875642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b377a5f-7b10-41d4-87d2-ac95fb5f6354 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:06.883026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 458/919] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:06.908061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 462/920] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:20:06 2022] POST /identity/v3/endpoints => generated 346 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:07.010295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5198811a-0544-4ba6-882b-10b1278ff167 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.010506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5198811a-0544-4ba6-882b-10b1278ff167 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.010775 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5198811a-0544-4ba6-882b-10b1278ff167 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.012530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 459/921] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:07.018149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646741-0cda-4ba8-a5c6-b9b7fa931bad None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.018434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646741-0cda-4ba8-a5c6-b9b7fa931bad None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.018708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646741-0cda-4ba8-a5c6-b9b7fa931bad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.038929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-1a646741-0cda-4ba8-a5c6-b9b7fa931bad None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:07.040182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1a646741-0cda-4ba8-a5c6-b9b7fa931bad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:07.120840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 463/922] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:07.158776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5198811a-0544-4ba6-882b-10b1278ff167 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:07.184817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:07.267778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5198811a-0544-4ba6-882b-10b1278ff167 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:07.307431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:07.308433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.308916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.309396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.310201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:07.311806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:07.321007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:07.322617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 06 04:20:07.323597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:07.326312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 460/923] 10.222.0.33 () {56 vars in 1132 bytes} [Thu Oct 6 04:20:07 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:07.331881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a646741-0cda-4ba8-a5c6-b9b7fa931bad None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:07.405262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:07.406641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:07.441544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a646741-0cda-4ba8-a5c6-b9b7fa931bad None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:07.480489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:07.481441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.481892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.482476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.483332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:07.484228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:07.486048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:07.496508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 464/924] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:07 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:07.502274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:07.515833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:07.533566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be0f190-27d2-4dd8-9f27-47e949a4a045 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:07.569814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:07.570785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.571289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.571828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.572633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:07.574248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:07.583789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:07.584818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:07.585675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:07.593722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 461/925] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:07 2022] GET /identity/v3/services/object-store => generated 93 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:07.594860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:07.627623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0cf8e4-eab9-459c-be16-9d5870950b4b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:07.670551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:07.671175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.671451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.671721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.672402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:07.672868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:07.674839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:07.677448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:07.687196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 465/926] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:07 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:07.688135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:07.713078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea7b7bf-f0b4-4381-9dd2-67923e1e5ec7 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:07.751760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:07.752772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.753265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.753741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.754685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:07.755326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:07.756932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:07.767668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 462/927] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:07 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:07.768796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:07.785410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:07.802901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3729ae8-6aa4-4ae9-b209-78c5a8ae87b4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:07.843158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:07.843765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.844043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.844316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.845262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:07.846633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:07.874985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 466/928] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:20:07 2022] POST /identity/v3/endpoints => generated 347 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:07.891132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64039c0-921a-4186-9758-efa33a6d1bbb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:07.938126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:07.938742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:07.939049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:07.939318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:07.940015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:07.940318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:07.941729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:07.948610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 463/929] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:07 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:07.954578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:08.021594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:08.080740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56999066-1d0a-4660-ad2b-d184e34e9308 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:08.116882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4638f21-7e58-482f-bb5d-02d2bd274d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:08.117855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4638f21-7e58-482f-bb5d-02d2bd274d15 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:08.118371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4638f21-7e58-482f-bb5d-02d2bd274d15 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:08.119383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4638f21-7e58-482f-bb5d-02d2bd274d15 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:08.120299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4638f21-7e58-482f-bb5d-02d2bd274d15 None admin] RBAC: Adding query filter params (interface=public, service_id=466d5b8a6c8a4af5a195bd732167aeba, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:08.120805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4638f21-7e58-482f-bb5d-02d2bd274d15 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:08.122345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4638f21-7e58-482f-bb5d-02d2bd274d15 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:08.128280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 467/930] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:20:07 2022] GET /identity/v3/endpoints?service_id=466d5b8a6c8a4af5a195bd732167aeba&interface=public®ion_id=RegionOne => generated 194 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:08.133723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:08.222554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:08.248216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d0aaee-dfc3-4320-a75a-5161fede3178 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:08.283877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd70499-c01e-4e2b-a992-23001180a44b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:08.284475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd70499-c01e-4e2b-a992-23001180a44b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:08.284747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd70499-c01e-4e2b-a992-23001180a44b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:08.285015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd70499-c01e-4e2b-a992-23001180a44b None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:08.285619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd70499-c01e-4e2b-a992-23001180a44b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:08.286995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd70499-c01e-4e2b-a992-23001180a44b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:08.293499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 464/931] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:20:08 2022] GET /identity/v3/services => generated 1497 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:20:08.563457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9566a3c-9414-4371-8051-5b5af3742afd None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:08.564450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9566a3c-9414-4371-8051-5b5af3742afd None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:08.564937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9566a3c-9414-4371-8051-5b5af3742afd None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:08.566757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 468/932] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:08.572729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe74159-0969-426f-92e2-190a5c9e6daa None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:08.573580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe74159-0969-426f-92e2-190a5c9e6daa None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:08.574064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe74159-0969-426f-92e2-190a5c9e6daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:08.592620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-bfe74159-0969-426f-92e2-190a5c9e6daa None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:08.594119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-bfe74159-0969-426f-92e2-190a5c9e6daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:08.671163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 465/933] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:08 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:08.937815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9566a3c-9414-4371-8051-5b5af3742afd None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.049284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9566a3c-9414-4371-8051-5b5af3742afd None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:09.074578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:09.091908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41104175-0fad-4b39-bed1-949205b8483f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:09.092542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41104175-0fad-4b39-bed1-949205b8483f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.092815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41104175-0fad-4b39-bed1-949205b8483f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.093080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41104175-0fad-4b39-bed1-949205b8483f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.104345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41104175-0fad-4b39-bed1-949205b8483f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-41104175-0fad-4b39-bed1-949205b8483f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:09.109029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:09.109986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:09.111077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 469/934] 10.222.0.33 () {56 vars in 1120 bytes} [Thu Oct 6 04:20:08 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:09.120177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe74159-0969-426f-92e2-190a5c9e6daa None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.191222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:09.240321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe74159-0969-426f-92e2-190a5c9e6daa None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:09.248903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42479f53-f2b8-4f09-b8a3-67fb38e6ebbd None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.249187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42479f53-f2b8-4f09-b8a3-67fb38e6ebbd None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.249462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42479f53-f2b8-4f09-b8a3-67fb38e6ebbd None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.251146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 470/935] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:09.256054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f50e48-3b0d-4fd8-bad7-da8c9dbfb9a5 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.256519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f50e48-3b0d-4fd8-bad7-da8c9dbfb9a5 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.256830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f50e48-3b0d-4fd8-bad7-da8c9dbfb9a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.279223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-71f50e48-3b0d-4fd8-bad7-da8c9dbfb9a5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:09.280549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-71f50e48-3b0d-4fd8-bad7-da8c9dbfb9a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:09.280889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:09.281662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.282107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.282538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.283389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:09.283700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:09.285064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:09.294394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 466/936] 10.222.0.33 () {56 vars in 1115 bytes} [Thu Oct 6 04:20:09 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:09.299800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.372472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 471/937] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:09.429530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ca39f4-b4cf-4474-845a-9f4226faa00e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:09.459966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:09.467206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:09.467836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.468115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.468389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.480575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:09.482025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:09.497330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:09.498291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:09.499705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:09.499705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:09.499705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.499705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 467/938] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:09.504078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f50e48-3b0d-4fd8-bad7-da8c9dbfb9a5 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.608461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f50e48-3b0d-4fd8-bad7-da8c9dbfb9a5 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:09.620148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:09.651049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:09.651687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.651965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.652239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.652894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:09.653195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:09.654561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:09.664908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.669987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 472/939] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:09.676036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.788890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab4790f-ebd0-4a6f-ab89-622575c43dc8 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:09.800906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba058a46-e7be-408c-8cb7-d7ecb96b99d1 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:09.823737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:09.824341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.824613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.824880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.825527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:09.827110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:09.829794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING keystone.server.flask.common [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 06 04:20:09.843541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:09.844475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:09.845072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:09.845538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:09.851935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.sql.core [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 06 04:20:09.851935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 06 04:20:09.851935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [parameters: {'id': '2e185f8b6de74140a84c55089932b7ac', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 06 04:20:09.851935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81400) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 06 04:20:09.855843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:09.857455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:20:09.869433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:09.870422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:20:09.871433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2e185f8b6de74140a84c55089932b7ac.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2e185f8b6de74140a84c55089932b7ac. Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result = self._query(query) Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application conn.query(q) Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 06 04:20:09.877116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result.read() Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 06 04:20:09.878090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application next(self.gen) Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application yield resource Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application next(self.gen) Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.session.rollback() Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.force_reraise() Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise self.value Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application session.commit() Oct 06 04:20:09.879087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._prepare_impl() Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.session.flush() Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._flush(objects) Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application compat.raise_( Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application flush_context.execute() Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 06 04:20:09.879986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rec.execute(self) Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 06 04:20:09.881837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result = self._query(query) Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application conn.query(q) Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result.read() Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 06 04:20:09.883881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [parameters: {'id': '2e185f8b6de74140a84c55089932b7ac', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:09.886276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:09.889630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2e185f8b6de74140a84c55089932b7ac. Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 468/940] 10.222.0.33 () {60 vars in 1140 bytes} [Thu Oct 6 04:20:09 2022] POST /identity/v3/projects => generated 262 bytes in 207 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:09.891839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.893236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 473/941] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:20:09 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:09.893236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:09.929733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:09.945397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:10.014742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ac1483-7646-419e-bd37-17b362b48bd6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.026000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5d093-1a71-4e38-948a-cd06483fb26d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.054601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.055497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.055927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.056849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.066595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.069944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.081570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.082455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.082889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.083328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.084231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:10.084691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.086513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:10.087205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:10.088332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:10.089430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:10.094994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 474/942] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:10.097845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:10.124511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 469/943] 10.222.0.33 () {56 vars in 1149 bytes} [Thu Oct 6 04:20:09 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:10.129768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:10.209859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:10.223668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fc0e36-d187-4a33-bd29-05b482880789 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.263137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:10.266066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eebc81-2ff8-415e-97a9-083e7d161f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.267198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eebc81-2ff8-415e-97a9-083e7d161f40 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.267687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eebc81-2ff8-415e-97a9-083e7d161f40 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.268163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eebc81-2ff8-415e-97a9-083e7d161f40 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.268747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac26987-a055-4986-8ca0-c557ccfc76fd None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.269248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eebc81-2ff8-415e-97a9-083e7d161f40 None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:10.269770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eebc81-2ff8-415e-97a9-083e7d161f40 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.271464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eebc81-2ff8-415e-97a9-083e7d161f40 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.299325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 475/944] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:20:10 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:10.304037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7702f738-2927-40cb-bce4-576cd60d4e61 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.304336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7702f738-2927-40cb-bce4-576cd60d4e61 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.304597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7702f738-2927-40cb-bce4-576cd60d4e61 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.306692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 476/945] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:10.308274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.311075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.311075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.311075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.313339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4da402-a2e8-4900-b8ed-868df74725bc None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.313625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4da402-a2e8-4900-b8ed-868df74725bc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.313896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4da402-a2e8-4900-b8ed-868df74725bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.322584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.324114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.333927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7a4da402-a2e8-4900-b8ed-868df74725bc None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:10.335665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7a4da402-a2e8-4900-b8ed-868df74725bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:10.343227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:10.344247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:10.345134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:10.345134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:10.345134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:10.345329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 470/946] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:10.351264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:10.431745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 477/947] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:10.483652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dd66e6-68ee-4b25-a811-2740f0751659 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.524768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afe4c15-ba4f-43b9-b2c7-739e8f508528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.525400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe4c15-ba4f-43b9-b2c7-739e8f508528 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.525682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe4c15-ba4f-43b9-b2c7-739e8f508528 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.525959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe4c15-ba4f-43b9-b2c7-739e8f508528 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.526609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afe4c15-ba4f-43b9-b2c7-739e8f508528 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:10.526916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afe4c15-ba4f-43b9-b2c7-739e8f508528 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.528359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afe4c15-ba4f-43b9-b2c7-739e8f508528 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.542818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 471/948] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:10.548216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4da402-a2e8-4900-b8ed-868df74725bc None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:10.609306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baef34-32fb-407d-95a0-1b6f2ead6961 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.609602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baef34-32fb-407d-95a0-1b6f2ead6961 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.609866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baef34-32fb-407d-95a0-1b6f2ead6961 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.611776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 472/949] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:10.616803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa26584d-341f-4a59-99a3-55b3da3919fc None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.617087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa26584d-341f-4a59-99a3-55b3da3919fc None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.617362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa26584d-341f-4a59-99a3-55b3da3919fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.637707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fa26584d-341f-4a59-99a3-55b3da3919fc None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:10.638600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:10.639257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fa26584d-341f-4a59-99a3-55b3da3919fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:10.641477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:10.661715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4da402-a2e8-4900-b8ed-868df74725bc None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.702156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.702810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.703135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.703407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.713373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.714828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:10.727535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:10.729099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:10.730024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 478/950] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:10.730516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:10.755167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 473/951] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:10.755167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa26584d-341f-4a59-99a3-55b3da3919fc None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:10.855060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a931ce3f-e624-45d3-be04-824a7c8202ac None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.882542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa26584d-341f-4a59-99a3-55b3da3919fc None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:10.900231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.900708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.901067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.901431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] PATH_INFO: `/v3/services/compute` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.902104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.903734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:10.914053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:10.915755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:10.917315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 479/952] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:10 2022] GET /identity/v3/services/compute => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:10.922340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:10.927251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:10.932534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:10.933920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:10.934601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:10.934885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:10.935620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:10.935929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:10.937500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:10.947909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:10.957066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 474/953] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:20:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:10.962785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:11.050705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa2661-ec32-4dd5-a6cd-bab76db5627b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:11.079240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edb9bb5-3f40-4fbb-869a-2b7e29a15d8f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:11.093776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:11.094394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:11.094661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:11.094936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:11.095594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] RBAC: Adding query filter params (name=compute) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:11.095899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:11.097266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:11.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 480/954] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:20:10 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:11.106522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:11.122129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:11.122796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:11.123169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:11.123447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/371190691e284514bafcba1bdd199fc6/roles/f1d374a285264fe38d7173e7d7603f55` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:11.125632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=371190691e284514bafcba1bdd199fc6, role_id=f1d374a285264fe38d7173e7d7603f55)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:11.127097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:11.155085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 475/955] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:20:10 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/371190691e284514bafcba1bdd199fc6/roles/f1d374a285264fe38d7173e7d7603f55 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:20:11.155085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:11.201937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:11.236161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:11.258690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6088d974-4d13-4874-a831-e4c8d09c508d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:11.301846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afb87bd-407f-4520-af2e-19065e9775f0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:11.302313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:11.303467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:11.303989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:11.304517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:11.305276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:11.307486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:11.318238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:11.319698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:11.320763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:11.323539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 481/956] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:11 2022] GET /identity/v3/services/object-store => generated 93 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:11.329024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:11.355024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:11.355645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:11.355924 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:11.356198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:11.356850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] RBAC: Adding query filter params (type=compute) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:11.357151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:11.358556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:11.364204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 476/957] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:20:11 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:11.450885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d668b50-d700-45e2-9ad5-c64d77a5a096 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:11.485876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:11.492017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:11.492805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:11.493076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:11.493999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:11.494638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:11.494933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:11.496339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:11.507682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:11.508132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 482/958] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:11 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:11.622560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe6ac4-4417-4b66-b0c0-15cfabcadf3d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:11.647813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:11.660664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129539d0-d838-4a50-9fca-851aa3b9ea3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:11.661273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129539d0-d838-4a50-9fca-851aa3b9ea3b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:11.661552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129539d0-d838-4a50-9fca-851aa3b9ea3b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:11.661823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129539d0-d838-4a50-9fca-851aa3b9ea3b None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:11.662455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129539d0-d838-4a50-9fca-851aa3b9ea3b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:11.662755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129539d0-d838-4a50-9fca-851aa3b9ea3b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:11.664224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129539d0-d838-4a50-9fca-851aa3b9ea3b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:11.670524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 477/959] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:11 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:11.675907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:11.789951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9b532c-a7b1-4917-aea0-1004ee56897f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:11.828440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de071da4-d862-49de-af44-aa1542baf69b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:11.829043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de071da4-d862-49de-af44-aa1542baf69b None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:11.829322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de071da4-d862-49de-af44-aa1542baf69b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:11.829594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de071da4-d862-49de-af44-aa1542baf69b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:11.830266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de071da4-d862-49de-af44-aa1542baf69b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:11.831669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de071da4-d862-49de-af44-aa1542baf69b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:11.852395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 483/960] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:20:11 2022] POST /identity/v3/endpoints => generated 349 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:12.651267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009db835-c5db-4cf8-9e50-8617012ea509 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:12.651798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009db835-c5db-4cf8-9e50-8617012ea509 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:12.651798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009db835-c5db-4cf8-9e50-8617012ea509 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:12.653296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 478/961] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:12.658659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27adc8fb-98ec-439e-a5cb-234157d1f511 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:12.659200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27adc8fb-98ec-439e-a5cb-234157d1f511 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:12.659704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27adc8fb-98ec-439e-a5cb-234157d1f511 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:12.678399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-27adc8fb-98ec-439e-a5cb-234157d1f511 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:12.680406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-27adc8fb-98ec-439e-a5cb-234157d1f511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:12.730210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:12.765759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 484/962] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:13.035406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009db835-c5db-4cf8-9e50-8617012ea509 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:13.145195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009db835-c5db-4cf8-9e50-8617012ea509 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:13.180871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:13.181579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:13.181805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:13.181988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:13.182637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:13.184088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:13.199744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.sql.core [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 06 04:20:13.199744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 06 04:20:13.199744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [parameters: {'user_id': 'dcd6aabc584e4f33896e4916e0d1ecd1', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 06 04:20:13.199744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81400) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result = self._query(query) Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application conn.query(q) Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result.read() Oct 06 04:20:13.207424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 06 04:20:13.208308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application next(self.gen) Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application yield resource Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application next(self.gen) Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.session.rollback() Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.force_reraise() Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise self.value Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application session.commit() Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 06 04:20:13.209269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._prepare_impl() Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.session.flush() Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._flush(objects) Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application compat.raise_( Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application flush_context.execute() Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rec.execute(self) Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 06 04:20:13.210177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 06 04:20:13.211122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result = self._query(query) Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application conn.query(q) Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application result.read() Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 06 04:20:13.212102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [parameters: {'user_id': 'dcd6aabc584e4f33896e4916e0d1ecd1', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:13.213651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 06 04:20:13.219829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 06 04:20:13.220859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 06 04:20:13.220859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 06 04:20:13.220859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 06 04:20:13.220859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 06 04:20:13.220859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 06 04:20:13.220859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:13.220859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 479/963] 10.222.0.33 () {60 vars in 1131 bytes} [Thu Oct 6 04:20:13 2022] POST /identity/v3/users => generated 160 bytes in 174 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:13.221254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27adc8fb-98ec-439e-a5cb-234157d1f511 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:13.326230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27adc8fb-98ec-439e-a5cb-234157d1f511 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:13.362605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f45d3d-6e4b-460d-9586-0f1d46750eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:13.363583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f45d3d-6e4b-460d-9586-0f1d46750eea None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:13.364059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f45d3d-6e4b-460d-9586-0f1d46750eea None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:13.364522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f45d3d-6e4b-460d-9586-0f1d46750eea None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:13.372543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f45d3d-6e4b-460d-9586-0f1d46750eea None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:13.374211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f45d3d-6e4b-460d-9586-0f1d46750eea None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-70f45d3d-6e4b-460d-9586-0f1d46750eea None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:20:13.387169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:13.388229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 06 04:20:13.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:13.391743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 485/964] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:20:13 2022] GET /identity/v3/users/cinder => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:13.398118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:13.456520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:13.512600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d8eae-c6c1-42c2-b601-c3821bb8d685 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:13.523299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f005578f-80c7-469c-a5cc-da5e9e3fd2e9 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:13.523723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f005578f-80c7-469c-a5cc-da5e9e3fd2e9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:13.524103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f005578f-80c7-469c-a5cc-da5e9e3fd2e9 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:13.527065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 486/965] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:13.532167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2439b62a-e5e8-4dc7-8a2a-9245df6a5deb None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:13.532683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2439b62a-e5e8-4dc7-8a2a-9245df6a5deb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:13.533265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2439b62a-e5e8-4dc7-8a2a-9245df6a5deb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:13.551200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480add2d-8f37-4f20-9226-c7e893557cf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:13.551813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480add2d-8f37-4f20-9226-c7e893557cf4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:13.552093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480add2d-8f37-4f20-9226-c7e893557cf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:13.552365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480add2d-8f37-4f20-9226-c7e893557cf4 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:13.553068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480add2d-8f37-4f20-9226-c7e893557cf4 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:13.553378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480add2d-8f37-4f20-9226-c7e893557cf4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:13.554880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2439b62a-e5e8-4dc7-8a2a-9245df6a5deb None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:13.555152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480add2d-8f37-4f20-9226-c7e893557cf4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:13.557164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2439b62a-e5e8-4dc7-8a2a-9245df6a5deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:13.591624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:13.600351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 480/966] 10.222.0.33 () {56 vars in 1101 bytes} [Thu Oct 6 04:20:13 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:13.641686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 487/967] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:13 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:13.727259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beff85f1-5ed0-4ce0-8ad1-c754ebe873db None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:13.728467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beff85f1-5ed0-4ce0-8ad1-c754ebe873db None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:13.728547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beff85f1-5ed0-4ce0-8ad1-c754ebe873db None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:13.730404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 481/968] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:13.735969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86d1bb-6252-4f8b-bca9-41a7615978af None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:13.736719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86d1bb-6252-4f8b-bca9-41a7615978af None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:13.736797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e86d1bb-6252-4f8b-bca9-41a7615978af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:13.758086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9e86d1bb-6252-4f8b-bca9-41a7615978af None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:13.760061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9e86d1bb-6252-4f8b-bca9-41a7615978af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:13.834562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 488/969] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:13 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:13.923887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beff85f1-5ed0-4ce0-8ad1-c754ebe873db None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.034090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beff85f1-5ed0-4ce0-8ad1-c754ebe873db None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.042680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.073156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad49b4c4-24e2-4b65-9738-da2cc0466f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.074173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad49b4c4-24e2-4b65-9738-da2cc0466f55 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.074648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad49b4c4-24e2-4b65-9738-da2cc0466f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.075147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad49b4c4-24e2-4b65-9738-da2cc0466f55 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.092668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad49b4c4-24e2-4b65-9738-da2cc0466f55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ad49b4c4-24e2-4b65-9738-da2cc0466f55 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:14.097906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:14.098856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:14.099761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:14.100425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be535a39-2493-4f02-b49d-d6dc148a5b6a None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.100425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be535a39-2493-4f02-b49d-d6dc148a5b6a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.100425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be535a39-2493-4f02-b49d-d6dc148a5b6a None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.101347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 489/970] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:14.103278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 482/971] 10.222.0.33 () {56 vars in 1120 bytes} [Thu Oct 6 04:20:13 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:14.104640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be535a39-2493-4f02-b49d-d6dc148a5b6a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.107150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c2ad86-0aff-42ea-a209-086c036017a3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.107494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c2ad86-0aff-42ea-a209-086c036017a3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.107718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c2ad86-0aff-42ea-a209-086c036017a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.126638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-91c2ad86-0aff-42ea-a209-086c036017a3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:14.128201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-91c2ad86-0aff-42ea-a209-086c036017a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:14.194846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 483/972] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:14.195661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c2ad86-0aff-42ea-a209-086c036017a3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.203922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.210067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be535a39-2493-4f02-b49d-d6dc148a5b6a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.250373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.251292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.251715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.252133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.252956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:14.253411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.254754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.269533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 490/973] 10.222.0.33 () {56 vars in 1115 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:14.270549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.302880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.310355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c2ad86-0aff-42ea-a209-086c036017a3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.350716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.351355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.351625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.351891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.352675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.356120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.374161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 484/974] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:20:14 2022] POST /identity/v3/services => generated 242 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:14.388640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae23f68-4ca5-46a1-b6b0-b6c2a2ca9f06 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.435867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.436518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.436790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.437055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.450423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.451809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:14.468179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:14.469263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:14.470200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:14.470200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:14.470200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:14.471314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 491/975] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:14.475122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.476934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.489383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.584774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa6012-376c-40ad-906b-15a83a89963a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.608111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffeb2d5-cb12-4888-a260-c37b31c67ff4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.623046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.623612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.623892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.624251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.624743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.626103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:14.635985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:14.637040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:14.638519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 492/976] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:14.642914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.648168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.648771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.649053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.649325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.650097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:14.650401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.651805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.670112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 485/977] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:14.671619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.692546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.722087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.758756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e7c506-da20-42c4-942a-4ecb9df9c5b6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.779451 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ea557-f965-42a7-810e-b332753e362f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.795117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.795844 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.796222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.796592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.797315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:14.797714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.799209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.804456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 493/978] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:14.810463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.816719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.817357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.817633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.817903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.825661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.827136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:20:14.839421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:14.840570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 06 04:20:14.841731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:14.845234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 486/979] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:14.846499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:14.928834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f48b156-305f-4680-bc3e-a3fc300bee6c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.932485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.958500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6305d57a-6fd7-412d-8aae-b7930d4186dc None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:14.966997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:14.967605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:14.967874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:14.968138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:14.968754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:14.969046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:14.970394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:14.972838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:14.976727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 494/980] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:14.983135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:15.006408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:15.007042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:15.007323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:15.007597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:15.008334 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:15.008642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:15.010762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:15.031801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 487/981] 10.222.0.33 () {56 vars in 1149 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:15.037253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:15.094087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ad9154-3bc8-441f-bfc9-95aa9fdbeb49 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:15.132164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:15.132768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:15.133040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:15.133316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:15.135011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] RBAC: Adding query filter params (interface=admin, service_id=466d5b8a6c8a4af5a195bd732167aeba, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:15.135317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:15.136677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:15.146246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 495/982] 10.222.0.33 () {56 vars in 1249 bytes} [Thu Oct 6 04:20:14 2022] GET /identity/v3/endpoints?service_id=466d5b8a6c8a4af5a195bd732167aeba&interface=admin®ion_id=RegionOne => generated 193 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:15.147239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:15.150429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc0584f-73cb-4c3a-998d-13acaad15155 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:15.185906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:15.186620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:15.186978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:15.187330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:15.190279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:15.215595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:15.262226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d322795-78fe-488b-8dd6-87200aace689 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:15.262422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:15.263904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:15.300360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:15.301065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:15.301370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:15.301602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:15.302195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:15.303581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:15.318995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 496/983] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:20:15 2022] GET /identity/v3/services => generated 1727 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:20:15.383228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 488/984] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:15 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:20:15.389572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:15.498170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91381d78-e836-4418-9c91-14a764c28c3b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:15.527216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:15.536110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:15.536743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:15.537021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:15.537290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:15.548527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:15.549958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:15.564879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:15.565860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:15.567488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:15.567488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:15.567488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:15.567488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 497/985] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:15.572164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:15.607981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:15.676873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ea0fa6-1b3b-48f0-957a-e72794209d88 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:15.712657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:15.713261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:15.713539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:15.713995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:15.714642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:15.714946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:15.716458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:15.732369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 489/986] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:15.734042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:15.838688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93176dbf-0820-4863-80a2-5e01b54ad9cf None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:15.872268 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:15.872890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:15.873167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:15.873435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:15.880952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:15.882368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:15.894620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:15.895592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:15.896551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:15.899276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 498/987] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:15.901827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:16.003074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:16.011025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f3c10-c84c-49f0-9be4-f9d2cd52fc1a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:16.047643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eda2ee6-c879-47fd-8aac-c7ed907bf26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:16.048255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda2ee6-c879-47fd-8aac-c7ed907bf26a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:16.048594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda2ee6-c879-47fd-8aac-c7ed907bf26a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:16.048872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda2ee6-c879-47fd-8aac-c7ed907bf26a None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:16.049612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda2ee6-c879-47fd-8aac-c7ed907bf26a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:16.049923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda2ee6-c879-47fd-8aac-c7ed907bf26a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:16.051390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda2ee6-c879-47fd-8aac-c7ed907bf26a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:16.069184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 490/988] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:20:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:16.070741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:16.114753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:16.174787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7918eba-fae3-40c2-91d3-dde8d0a5cd64 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:16.218743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26fdc37-2a14-408e-a606-8d19886c66f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:16.219391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fdc37-2a14-408e-a606-8d19886c66f8 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:16.219669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fdc37-2a14-408e-a606-8d19886c66f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:16.219941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26fdc37-2a14-408e-a606-8d19886c66f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:16.220660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fdc37-2a14-408e-a606-8d19886c66f8 None admin] RBAC: Adding query filter params (role.id=f1d374a285264fe38d7173e7d7603f55, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=371190691e284514bafcba1bdd199fc6) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:16.220971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fdc37-2a14-408e-a606-8d19886c66f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:16.222387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26fdc37-2a14-408e-a606-8d19886c66f8 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:16.239047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 499/989] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:16 2022] GET /identity/v3/role_assignments?user.id=371190691e284514bafcba1bdd199fc6&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=f1d374a285264fe38d7173e7d7603f55 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:16.548921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c491d-e247-4ff0-a555-b5338d92c7c5 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:16.549264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c491d-e247-4ff0-a555-b5338d92c7c5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:16.549384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c491d-e247-4ff0-a555-b5338d92c7c5 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:16.550867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 491/990] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:16.556479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b2be6-12bc-4ae7-9112-0c1ce24d3e95 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:16.557753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b2be6-12bc-4ae7-9112-0c1ce24d3e95 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:16.557753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b2be6-12bc-4ae7-9112-0c1ce24d3e95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:16.575557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b68b2be6-12bc-4ae7-9112-0c1ce24d3e95 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:16.576741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b68b2be6-12bc-4ae7-9112-0c1ce24d3e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:16.646600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 500/991] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:16 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:16.911120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c491d-e247-4ff0-a555-b5338d92c7c5 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:17.015423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c491d-e247-4ff0-a555-b5338d92c7c5 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:17.050559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:17.051211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.051482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.051747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.052330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:17.053693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:17.063101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:17.064291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:17.065297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 492/992] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:16 2022] GET /identity/v3/services/compute => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:17.069963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68b2be6-12bc-4ae7-9112-0c1ce24d3e95 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:17.103707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:17.173739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68b2be6-12bc-4ae7-9112-0c1ce24d3e95 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:17.208607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:17.209206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.211056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.211056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.211056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:17.211056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:17.212834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:17.224200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 501/993] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:20:17 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:17.225175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:17.246212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:17.329628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abdf97-d6b3-48b3-8e3d-d41ede578247 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:17.365642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af107aac-8d03-44a6-a03d-2b6269acdcaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:17.366272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af107aac-8d03-44a6-a03d-2b6269acdcaa None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.366547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af107aac-8d03-44a6-a03d-2b6269acdcaa None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.366820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af107aac-8d03-44a6-a03d-2b6269acdcaa None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.367498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af107aac-8d03-44a6-a03d-2b6269acdcaa None admin] RBAC: Adding query filter params (type=compute) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:17.367801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af107aac-8d03-44a6-a03d-2b6269acdcaa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:17.369319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af107aac-8d03-44a6-a03d-2b6269acdcaa None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:17.380027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 493/994] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:20:17 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:17.381455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:17.477242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5d206-43b8-4607-adda-00bbe1de8d3b None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.477450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5d206-43b8-4607-adda-00bbe1de8d3b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.477760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5d206-43b8-4607-adda-00bbe1de8d3b None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.479207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 494/995] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:17.484864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ab1c4-f403-4e15-8921-f848fa5bcb27 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.485143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ab1c4-f403-4e15-8921-f848fa5bcb27 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.485414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ab1c4-f403-4e15-8921-f848fa5bcb27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.485648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:17.489991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d68f32f-ec55-4bd2-90aa-a3786b69764b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:17.503832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-458ab1c4-f403-4e15-8921-f848fa5bcb27 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:17.505008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-458ab1c4-f403-4e15-8921-f848fa5bcb27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:17.528080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:17.528833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.529295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.529689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.530451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] RBAC: Adding query filter params (interface=public, service_id=b9626857e26e4f00900de65dc514a8fb, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:17.530755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:17.532244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:17.538083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 502/996] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:20:17 2022] GET /identity/v3/endpoints?service_id=b9626857e26e4f00900de65dc514a8fb&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:17.543761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:17.579335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 495/997] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:17 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:17.645192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb5bb63-e11b-4a73-8052-b2b7584f76dd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:17.678916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:17.679522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.679796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.680062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.680771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:17.682104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:17.688472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 503/998] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:20:17 2022] GET /identity/v3/services => generated 1727 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:20:17.844977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ab1c4-f403-4e15-8921-f848fa5bcb27 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:17.861804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:17.950730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ab1c4-f403-4e15-8921-f848fa5bcb27 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:17.984118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:17.984741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:17.985012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:17.985278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:17.985853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:17.987493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:17.996578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:17.997643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:17.998525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 496/999] 10.222.0.33 () {56 vars in 1126 bytes} [Thu Oct 6 04:20:17 2022] GET /identity/v3/services/object-store => generated 93 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:18.004812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:18.033031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:18.106560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0218b1f7-b97b-4d10-ad3e-2bee5f6f6347 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:18.140954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:18.141548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:18.141819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:18.142083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:18.142691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:18.143006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:18.144340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:18.149168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 504/1000] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:18 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:18.154723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:18.246203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:18.270217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e8e034-5b9b-4641-b886-98effa8e4c16 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:18.311876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1582193-c3a1-4cf5-8df3-ea35438edef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:18.312648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1582193-c3a1-4cf5-8df3-ea35438edef1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:18.312764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1582193-c3a1-4cf5-8df3-ea35438edef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:18.313011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1582193-c3a1-4cf5-8df3-ea35438edef1 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:18.313641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1582193-c3a1-4cf5-8df3-ea35438edef1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:18.313939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1582193-c3a1-4cf5-8df3-ea35438edef1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:18.315396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1582193-c3a1-4cf5-8df3-ea35438edef1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:18.320746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 497/1001] 10.222.0.33 () {56 vars in 1122 bytes} [Thu Oct 6 04:20:18 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:18.326709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:18.421437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:18.432989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad28d7-7cb4-443b-bad9-53a15c8bf4c1 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:18.459214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54f9782-6f11-4218-ade0-83108654dcca None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:18.459492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54f9782-6f11-4218-ade0-83108654dcca None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:18.459762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54f9782-6f11-4218-ade0-83108654dcca None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:18.461099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 498/1002] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:18.466566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ba2ff2-2a2d-4a29-af16-8d0cb55fbe75 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:18.466847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ba2ff2-2a2d-4a29-af16-8d0cb55fbe75 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:18.467151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ba2ff2-2a2d-4a29-af16-8d0cb55fbe75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:18.475532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:18.476141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:18.476417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:18.476687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:18.477423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:18.479317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:18.485969 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-45ba2ff2-2a2d-4a29-af16-8d0cb55fbe75 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:18.487519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-45ba2ff2-2a2d-4a29-af16-8d0cb55fbe75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:18.497463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 505/1003] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:20:18 2022] POST /identity/v3/endpoints => generated 325 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:18.575174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 499/1004] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:18 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:18.838771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:18.936639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8abd11-25f5-4c29-9c33-b3553dbeacce None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:18.971566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:18.972201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:18.972480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:18.972754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] PATH_INFO: `/v3/services/image` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:18.973335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:18.974696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:18.984786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:18.985823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:18.986994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 506/1005] 10.222.0.33 () {56 vars in 1105 bytes} [Thu Oct 6 04:20:18 2022] GET /identity/v3/services/image => generated 86 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:18.991913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ba2ff2-2a2d-4a29-af16-8d0cb55fbe75 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:19.058345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:19.093025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ba2ff2-2a2d-4a29-af16-8d0cb55fbe75 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:19.127303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ece3fa2-5864-428a-a3e0-2f3d888dacff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:19.127900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ece3fa2-5864-428a-a3e0-2f3d888dacff None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:19.128172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ece3fa2-5864-428a-a3e0-2f3d888dacff None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:19.128436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ece3fa2-5864-428a-a3e0-2f3d888dacff None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:19.129052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ece3fa2-5864-428a-a3e0-2f3d888dacff None admin] RBAC: Adding query filter params (name=image) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:19.129348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ece3fa2-5864-428a-a3e0-2f3d888dacff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:19.130695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ece3fa2-5864-428a-a3e0-2f3d888dacff None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:19.140256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 500/1006] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:18 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:19.141449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:19.157992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:19.250796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea15f986-a2b8-4173-84d3-32f8bd518b9e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:19.285597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae5f4b-b443-4f06-b60d-27e49d51349b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:19.286198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae5f4b-b443-4f06-b60d-27e49d51349b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:19.286468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae5f4b-b443-4f06-b60d-27e49d51349b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:19.286733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae5f4b-b443-4f06-b60d-27e49d51349b None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:19.287373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae5f4b-b443-4f06-b60d-27e49d51349b None admin] RBAC: Adding query filter params (type=image) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:19.287670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae5f4b-b443-4f06-b60d-27e49d51349b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:19.289005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae5f4b-b443-4f06-b60d-27e49d51349b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:19.294470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 507/1007] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:19 2022] GET /identity/v3/services?type=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:19.906158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df02db7-cea3-4958-ad7d-97ebfcd547e0 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:19.906512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df02db7-cea3-4958-ad7d-97ebfcd547e0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:19.906714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df02db7-cea3-4958-ad7d-97ebfcd547e0 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:19.908124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 501/1008] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:19.913312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c288194-5caa-437b-afb2-bc46649ab2be None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:19.913596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c288194-5caa-437b-afb2-bc46649ab2be None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:19.913869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c288194-5caa-437b-afb2-bc46649ab2be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:19.932140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4c288194-5caa-437b-afb2-bc46649ab2be None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:19.933312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4c288194-5caa-437b-afb2-bc46649ab2be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:20.000555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 508/1009] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:19 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:20.270186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df02db7-cea3-4958-ad7d-97ebfcd547e0 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:20.373794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df02db7-cea3-4958-ad7d-97ebfcd547e0 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:20.403536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:20.410939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:20.411593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:20.411866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:20.412131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:20.412705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:20.414068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:20.423711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:20.424730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:20.425593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 502/1010] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:20.432100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c288194-5caa-437b-afb2-bc46649ab2be None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:20.442369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:20.532979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c288194-5caa-437b-afb2-bc46649ab2be None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:20.566116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09c8d93-82b6-4082-b8b3-edda5f1787fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:20.566710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09c8d93-82b6-4082-b8b3-edda5f1787fd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:20.567006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09c8d93-82b6-4082-b8b3-edda5f1787fd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:20.567281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09c8d93-82b6-4082-b8b3-edda5f1787fd None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:20.567903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09c8d93-82b6-4082-b8b3-edda5f1787fd None admin] RBAC: Adding query filter params (name=compute) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:20.568200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09c8d93-82b6-4082-b8b3-edda5f1787fd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:20.569559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09c8d93-82b6-4082-b8b3-edda5f1787fd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:20.580171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 509/1011] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:20:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:20.581405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:20.686549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f7f80-5965-48e3-800f-de2c28620bd0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:20.699966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827c2a76-045e-4bc6-aa80-172048dd002b None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:20.700288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827c2a76-045e-4bc6-aa80-172048dd002b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:20.700513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827c2a76-045e-4bc6-aa80-172048dd002b None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:20.702066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 510/1012] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:20.707493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac533e-f663-408e-95ba-c4be138d6788 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:20.707799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac533e-f663-408e-95ba-c4be138d6788 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:20.708050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac533e-f663-408e-95ba-c4be138d6788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:20.724598 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:20.725204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:20.725479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:20.725760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:20.726382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] RBAC: Adding query filter params (type=compute) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:20.726680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:20.728336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:20.729860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a8ac533e-f663-408e-95ba-c4be138d6788 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:20.731081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a8ac533e-f663-408e-95ba-c4be138d6788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:20.733376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 503/1013] 10.222.0.33 () {56 vars in 1112 bytes} [Thu Oct 6 04:20:20 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:20.739183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:20.799314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:20.802844 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 511/1014] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:20 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:20.846428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227ca8d-c4e3-4623-9864-b3750f6971fb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:20.881883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:20.882481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:20.882757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:20.883049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:20.883806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:20.885166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:20.910992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 504/1015] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:20:20 2022] POST /identity/v3/endpoints => generated 334 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:20:21.078239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac533e-f663-408e-95ba-c4be138d6788 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:21.080460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:21.189951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac533e-f663-408e-95ba-c4be138d6788 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:21.232750 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c47b1f-b0cd-41e7-b269-6d8f9eb36caa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:21.233372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c47b1f-b0cd-41e7-b269-6d8f9eb36caa None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:21.233643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c47b1f-b0cd-41e7-b269-6d8f9eb36caa None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:21.233908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c47b1f-b0cd-41e7-b269-6d8f9eb36caa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:21.235033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c47b1f-b0cd-41e7-b269-6d8f9eb36caa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:21.236698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c47b1f-b0cd-41e7-b269-6d8f9eb36caa None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:21.238238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 512/1016] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:21.244049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:21.350953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55f3216-b657-484f-bbd7-e05a51b5ff07 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:21.366873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:21.390072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232d554d-e65e-4d14-a6ed-16dda5f5e9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:21.390671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d554d-e65e-4d14-a6ed-16dda5f5e9a6 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:21.390942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d554d-e65e-4d14-a6ed-16dda5f5e9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:21.391237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232d554d-e65e-4d14-a6ed-16dda5f5e9a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:21.391878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232d554d-e65e-4d14-a6ed-16dda5f5e9a6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:21.393292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232d554d-e65e-4d14-a6ed-16dda5f5e9a6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:21.422658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 505/1017] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:20:21 2022] POST /identity/v3/projects => generated 311 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:20:21.458744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d31093-a286-4b33-b7ca-5aa6d5fbaa86 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:21.459354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d31093-a286-4b33-b7ca-5aa6d5fbaa86 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:21.459354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d31093-a286-4b33-b7ca-5aa6d5fbaa86 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:21.460846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 513/1018] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:21.466640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bf1397-331e-48b8-9224-618b5575891f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:21.466941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bf1397-331e-48b8-9224-618b5575891f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:21.467236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bf1397-331e-48b8-9224-618b5575891f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:21.486860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f3bf1397-331e-48b8-9224-618b5575891f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:21.488121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f3bf1397-331e-48b8-9224-618b5575891f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:21.564190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 506/1019] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:21 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:21.828426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d31093-a286-4b33-b7ca-5aa6d5fbaa86 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:21.896181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:21.933781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d31093-a286-4b33-b7ca-5aa6d5fbaa86 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:21.969348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f948e7-9fd5-452d-a5d0-6135a56ec1b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:21.969933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f948e7-9fd5-452d-a5d0-6135a56ec1b6 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:21.970204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f948e7-9fd5-452d-a5d0-6135a56ec1b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:21.970466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f948e7-9fd5-452d-a5d0-6135a56ec1b6 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:21.971137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f948e7-9fd5-452d-a5d0-6135a56ec1b6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:21.972470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f948e7-9fd5-452d-a5d0-6135a56ec1b6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:21.988778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 514/1020] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:20:21 2022] POST /identity/v3/services => generated 242 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:23.243371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a133ed97-8db0-48f6-9f8e-6a814161c6e3 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:23.243953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a133ed97-8db0-48f6-9f8e-6a814161c6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:23.243953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a133ed97-8db0-48f6-9f8e-6a814161c6e3 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:23.245459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 507/1021] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:23.250668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96791b-232d-4cde-8c3a-e7afdc031793 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:23.251085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96791b-232d-4cde-8c3a-e7afdc031793 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:23.251251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96791b-232d-4cde-8c3a-e7afdc031793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:23.270195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-5d96791b-232d-4cde-8c3a-e7afdc031793 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:23.271407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5d96791b-232d-4cde-8c3a-e7afdc031793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:23.344933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 515/1022] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:23 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:23.618718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a133ed97-8db0-48f6-9f8e-6a814161c6e3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:23.720835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356518d-3aba-4c74-82e9-72ad18e342b8 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:23.721128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356518d-3aba-4c74-82e9-72ad18e342b8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:23.722223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356518d-3aba-4c74-82e9-72ad18e342b8 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:23.722333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:23.723004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 516/1023] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:23.729731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b084679-96c0-400b-aa24-dd2c95c5ab0e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:23.729996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b084679-96c0-400b-aa24-dd2c95c5ab0e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:23.730282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b084679-96c0-400b-aa24-dd2c95c5ab0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:23.732518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a133ed97-8db0-48f6-9f8e-6a814161c6e3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:23.749746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9b084679-96c0-400b-aa24-dd2c95c5ab0e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:23.750936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9b084679-96c0-400b-aa24-dd2c95c5ab0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:23.771684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:23.772315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:23.772594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:23.772866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:23.787983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:23.792981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:23.793970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:23.794840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 508/1024] 10.222.0.33 () {56 vars in 1120 bytes} [Thu Oct 6 04:20:23 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:23.800123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:23.811022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:23.838877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 517/1025] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:23 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:23.912904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9f0dd-f001-412b-a3b1-9a8502fa5feb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:23.949554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:23.950157 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:23.950435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:23.950708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:23.951428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:23.951732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:23.953099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:23.967327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 509/1026] 10.222.0.33 () {56 vars in 1115 bytes} [Thu Oct 6 04:20:23 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:23.968792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b084679-96c0-400b-aa24-dd2c95c5ab0e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.078311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b084679-96c0-400b-aa24-dd2c95c5ab0e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.115311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.115936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.116331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.116589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.124788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.129775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:24.131334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:24.146932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:24.147207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:24.148168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:24.154394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:24.154394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:24.154394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:24.154394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 518/1027] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:24.156194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.197614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:24.234459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af36370-8dfe-4aae-bc7c-a44c1aac7cf6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.267422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf297181-6724-4055-94a6-c726342321d9 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.269147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5dad9a-5b79-4a0d-8b5b-00cb4b0bdfc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.269767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5dad9a-5b79-4a0d-8b5b-00cb4b0bdfc1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.270041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5dad9a-5b79-4a0d-8b5b-00cb4b0bdfc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.270306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5dad9a-5b79-4a0d-8b5b-00cb4b0bdfc1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.271446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5dad9a-5b79-4a0d-8b5b-00cb4b0bdfc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:24.272864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5dad9a-5b79-4a0d-8b5b-00cb4b0bdfc1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:24.275168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 510/1028] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:24.276271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206c8bb3-4cda-4c48-b135-e447a954ec00 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.276881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206c8bb3-4cda-4c48-b135-e447a954ec00 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.276881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206c8bb3-4cda-4c48-b135-e447a954ec00 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.278103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 511/1029] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:24.281208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206c8bb3-4cda-4c48-b135-e447a954ec00 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.304148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb233df-1e68-436c-a554-18ccb2f8b00d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.304747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb233df-1e68-436c-a554-18ccb2f8b00d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.305025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb233df-1e68-436c-a554-18ccb2f8b00d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.305334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb233df-1e68-436c-a554-18ccb2f8b00d None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.306128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb233df-1e68-436c-a554-18ccb2f8b00d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:24.306433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb233df-1e68-436c-a554-18ccb2f8b00d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:24.307823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb233df-1e68-436c-a554-18ccb2f8b00d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:24.319241 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 519/1030] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:24.320475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b0515-f702-4337-886c-98738e4f1afc None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.320764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b0515-f702-4337-886c-98738e4f1afc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.321622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b0515-f702-4337-886c-98738e4f1afc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.340475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8b0b0515-f702-4337-886c-98738e4f1afc None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:24.341659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8b0b0515-f702-4337-886c-98738e4f1afc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:24.366467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:24.381997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206c8bb3-4cda-4c48-b135-e447a954ec00 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.408231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 520/1031] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:24 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:24.408900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0b0515-f702-4337-886c-98738e4f1afc None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.421838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.422439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.422712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.423009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.423641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:24.425062 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:24.434035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:24.459043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 512/1032] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:20:24 2022] POST /identity/v3/users => generated 285 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:20:24.520700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0b0515-f702-4337-886c-98738e4f1afc None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.559876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.560525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.560804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.561075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.569319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:24.570792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:20:24.583847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:24.584912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:24.586458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 521/1033] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:20:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:24.591536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.683251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.705780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de28ed02-b9df-42c8-b188-750a81a438be None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.743387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.744251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.744522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.745234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.746238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:24.746554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:24.748516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:24.765106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:24.769535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 513/1034] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:20:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:24.776206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.781986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:24.799474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c669667d-943c-4e06-add0-4ce9b9c2d449 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.834983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.835616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.835897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.836154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] PATH_INFO: `/v3/services/image` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.837252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:24.838612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:24.847462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:24.848525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:24.849411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 522/1035] 10.222.0.33 () {56 vars in 1105 bytes} [Thu Oct 6 04:20:24 2022] GET /identity/v3/services/image => generated 86 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:24.854818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:24.894255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109ec1c0-042c-48c9-8f85-64844b91dce7 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:24.930347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:24.931297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:24.931579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:24.931848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:24.970850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ef5aba-9051-41d6-aafc-e6f6b42aacb4 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.001285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.002758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.004828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:25.011882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.012479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.012829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.013061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.013617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] RBAC: Adding query filter params (name=image) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:25.013911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.015267 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.020344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 523/1036] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:24 2022] GET /identity/v3/services?name=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:25.027045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.044999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:25.126937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 514/1037] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:24 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 06 04:20:25.128010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.152322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7a405c-1d22-4255-a43d-233615515f01 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.192398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.193275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.193693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.194114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.194740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] RBAC: Adding query filter params (type=image) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:25.195061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.196448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.202109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 524/1038] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/services?type=image => generated 350 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:25.208213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.252153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:25.255743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b8dd41-a4d4-432a-8a8f-bf431b76c08c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.295524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.296199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:25.296789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.297183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.297349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.308235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.310108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.321037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9043dc36-8e3f-47f0-812f-7fcd63072d60 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:25.328161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:25.329224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:25.330315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:25.330315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:25.330315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:25.336390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 515/1039] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:25.337947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.359422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.360018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.360664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.360664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.361233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] RBAC: Adding query filter params (interface=public, service_id=b6612acb75f54de5b274f37238c7872d, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:25.361783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.363157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.369290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 525/1040] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/endpoints?service_id=b6612acb75f54de5b274f37238c7872d&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:25.374801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.444572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce426517-92ce-4165-b956-65bacff895fd None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.480939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.481545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.481825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.482105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.482739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:25.483082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.484458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.490067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55673b3c-917b-42a9-8915-4d7f348099da None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.498326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 516/1041] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:25.504166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.523696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:25.529748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.530744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.531185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.531464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.532127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.533667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.551792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 526/1042] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/services => generated 1957 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:20:25.640448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc72bdcb-d777-4e26-9b20-83b9cde42dd5 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.677063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.677705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.677981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.678255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.686382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.687901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:25.700085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:25.701140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:25.707396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 517/1043] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:25.707932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.713152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:25.818248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72bc48a-aa8d-4cf4-933f-4da6df119c0f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:25.853853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb2b301-6880-4e49-8556-ff0c025e9029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:25.854455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2b301-6880-4e49-8556-ff0c025e9029 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:25.854735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2b301-6880-4e49-8556-ff0c025e9029 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:25.855033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb2b301-6880-4e49-8556-ff0c025e9029 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:25.855723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb2b301-6880-4e49-8556-ff0c025e9029 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:25.856061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb2b301-6880-4e49-8556-ff0c025e9029 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:25.857443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb2b301-6880-4e49-8556-ff0c025e9029 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:25.875420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 527/1044] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:25.877042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:25.974449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:26.005856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d50f8d6-3a58-4922-b076-a845282f89f4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:26.041535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebce7db-0af9-42e9-a5f3-dfc5c8f2f18d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:26.042182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebce7db-0af9-42e9-a5f3-dfc5c8f2f18d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:26.042465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebce7db-0af9-42e9-a5f3-dfc5c8f2f18d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:26.042735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebce7db-0af9-42e9-a5f3-dfc5c8f2f18d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:26.043490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebce7db-0af9-42e9-a5f3-dfc5c8f2f18d None admin] RBAC: Adding query filter params (role.id=f1d374a285264fe38d7173e7d7603f55, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=c8d2db883de44964b6e15b85a97726b5) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:26.043796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebce7db-0af9-42e9-a5f3-dfc5c8f2f18d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:26.045217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebce7db-0af9-42e9-a5f3-dfc5c8f2f18d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:26.059241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 518/1045] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:25 2022] GET /identity/v3/role_assignments?user.id=c8d2db883de44964b6e15b85a97726b5&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=f1d374a285264fe38d7173e7d7603f55 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:26.707799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e469ff-152f-4fc0-be8d-ff48003cdd9e None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:26.708332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e469ff-152f-4fc0-be8d-ff48003cdd9e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:26.708332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e469ff-152f-4fc0-be8d-ff48003cdd9e None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:26.709664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 528/1046] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:26.715727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12977ca8-15e6-447e-8090-1d12c723b2d4 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:26.716158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12977ca8-15e6-447e-8090-1d12c723b2d4 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:26.716261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12977ca8-15e6-447e-8090-1d12c723b2d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:26.735396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-12977ca8-15e6-447e-8090-1d12c723b2d4 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:26.736829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-12977ca8-15e6-447e-8090-1d12c723b2d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:26.804981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 519/1047] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:26 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:27.087179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e469ff-152f-4fc0-be8d-ff48003cdd9e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:27.151893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:27.193945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e469ff-152f-4fc0-be8d-ff48003cdd9e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:27.238549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:27.239376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:27.239655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:27.239921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:27.251512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:27.256173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:27.257111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:27.259646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 529/1048] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:20:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:27.264006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12977ca8-15e6-447e-8090-1d12c723b2d4 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:27.331919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:27.371662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12977ca8-15e6-447e-8090-1d12c723b2d4 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:27.406270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:27.406926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:27.407229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:27.407495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:27.408144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:27.408439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:27.409793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:27.423388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 520/1049] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:20:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:27.424931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:27.529183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e49c09-c52f-4686-a165-11b54a35868a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:27.552805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:27.564722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e18bf83-13fb-495a-a7e9-638c352afb8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:27.565341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e18bf83-13fb-495a-a7e9-638c352afb8f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:27.565612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e18bf83-13fb-495a-a7e9-638c352afb8f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:27.565874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e18bf83-13fb-495a-a7e9-638c352afb8f None admin] PATH_INFO: `/v3/users/3bcfea5e66e7464691dfa58e11a8c485` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:27.580189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e18bf83-13fb-495a-a7e9-638c352afb8f None admin] RBAC: Authorizing `identity:get_user(user_id=3bcfea5e66e7464691dfa58e11a8c485)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:27.581610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e18bf83-13fb-495a-a7e9-638c352afb8f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:27.588588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 530/1050] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:27 2022] GET /identity/v3/users/3bcfea5e66e7464691dfa58e11a8c485 => generated 285 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:27.590088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:27.702376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8555c0c0-6f6c-4b27-8432-7d992abbad0c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:27.730610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:27.738036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a72bc8-1234-484e-ab97-339783b48350 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:27.738477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a72bc8-1234-484e-ab97-339783b48350 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:27.738568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a72bc8-1234-484e-ab97-339783b48350 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:27.740209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 531/1051] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:27.741375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:27.741982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:27.742262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:27.743146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:27.745335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55082d12-32dd-4080-9bc0-f2bf380bee69 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:27.745615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55082d12-32dd-4080-9bc0-f2bf380bee69 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:27.745885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55082d12-32dd-4080-9bc0-f2bf380bee69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:27.763835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-55082d12-32dd-4080-9bc0-f2bf380bee69 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:27.765008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-55082d12-32dd-4080-9bc0-f2bf380bee69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:27.820179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:27.822491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:27.832504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 532/1052] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:27 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:27.932672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 521/1053] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:27 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 06 04:20:27.934467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55082d12-32dd-4080-9bc0-f2bf380bee69 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.035925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55082d12-32dd-4080-9bc0-f2bf380bee69 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.048910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:28.071396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.072020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.072296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.072565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.073678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] RBAC: Authorizing `identity:get_project(project_id=84ce0d06721640798467c802ae991b78)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:28.075139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:28.076911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 533/1054] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:20:27 2022] GET /identity/v3/projects/84ce0d06721640798467c802ae991b78 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:28.082893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.098076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.116342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:28.205191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72bd18f-568a-48ab-b1f0-82412a6d8aad None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.207609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b038e8a-4e84-43e1-b3de-75658003c8ae None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.245605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90f5d6c-af55-4658-bfe0-6e4390ce25f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.246204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f5d6c-af55-4658-bfe0-6e4390ce25f3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.246475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f5d6c-af55-4658-bfe0-6e4390ce25f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.246749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f5d6c-af55-4658-bfe0-6e4390ce25f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.247497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90f5d6c-af55-4658-bfe0-6e4390ce25f3 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=84ce0d06721640798467c802ae991b78, user.id=3bcfea5e66e7464691dfa58e11a8c485) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:28.247802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90f5d6c-af55-4658-bfe0-6e4390ce25f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:28.249230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90f5d6c-af55-4658-bfe0-6e4390ce25f3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:28.250807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.251453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.251726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.251997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] PATH_INFO: `/v3/services/image` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.252561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:28.253877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:28.254638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 522/1055] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:28 2022] GET /identity/v3/role_assignments?user.id=3bcfea5e66e7464691dfa58e11a8c485&scope.project.id=84ce0d06721640798467c802ae991b78&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:28.255839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48dff3e-a743-4d1b-8c3a-9f3433e69024 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.256126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48dff3e-a743-4d1b-8c3a-9f3433e69024 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.256398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48dff3e-a743-4d1b-8c3a-9f3433e69024 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.257703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 523/1056] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:28.263251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:28.264320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 06 04:20:28.265214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:28.266525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd397d9-db2f-47a6-a6d8-d3feb75a2667 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.266525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd397d9-db2f-47a6-a6d8-d3feb75a2667 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.266525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd397d9-db2f-47a6-a6d8-d3feb75a2667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.267197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 534/1057] 10.222.0.33 () {56 vars in 1105 bytes} [Thu Oct 6 04:20:28 2022] GET /identity/v3/services/image => generated 86 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:28.270707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.283470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5fd397d9-db2f-47a6-a6d8-d3feb75a2667 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:28.284700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5fd397d9-db2f-47a6-a6d8-d3feb75a2667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:28.330134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:28.365122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 524/1058] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:28 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:28.394908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d03a089-b242-44da-a1bd-c77b9f33231c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.432011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.432615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.432883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.433156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.433791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] RBAC: Adding query filter params (name=image) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:28.434094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:28.435483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:28.441345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 535/1059] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:28 2022] GET /identity/v3/services?name=image => generated 122 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:28.447662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd397d9-db2f-47a6-a6d8-d3feb75a2667 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.472147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:28.552652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd397d9-db2f-47a6-a6d8-d3feb75a2667 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.588005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.588528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.588806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.589073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.589696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] RBAC: Adding query filter params (type=image) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:28.590049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:28.591444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:28.600957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 525/1060] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:28 2022] GET /identity/v3/services?type=image => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:28.602747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.631788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.709085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5dab9a-874c-4caf-867b-2ecc134798bd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.719915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:28.722233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:28.741684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763e997-84d8-49b2-89a7-69fad30ee70a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.750375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.750997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.751274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.751593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.752630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:28.754284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:28.782993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 536/1061] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:20:28 2022] POST /identity/v3/endpoints => generated 327 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:20:28.791003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.791710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.791989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.792260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.804435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:28.810400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:28.811457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:28.816015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 526/1062] 10.222.0.33 () {56 vars in 1120 bytes} [Thu Oct 6 04:20:28 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:28.818285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:28.935681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85e0a76-d7d8-41f7-a2e1-5e7f967aac24 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:28.976103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:28.977657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:28.978180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:28.978662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:28.979549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:28.979994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:28.981515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:28.991271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 537/1063] 10.222.0.33 () {56 vars in 1115 bytes} [Thu Oct 6 04:20:28 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:28.997269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:29.103006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba09b5dc-e327-43bb-8e28-3a6fa4fc370b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:29.144659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:29.145284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:29.145557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:29.145824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:29.156432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:29.157899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:29.159848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:29.176018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:29.177099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:29.178049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:29.178049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:29.178049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:29.179035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 527/1064] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:29.183689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:29.216074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:29.293511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b25fec-09de-4d51-a2a8-f1c4f1c13c51 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:29.328350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:29.329257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:29.329781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:29.330266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:29.331128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:29.331645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:29.333358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:29.346254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 538/1065] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:29.352785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:29.455667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3b417f-c66b-4db1-a067-f86aeaa81fdd None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:29.491937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:29.493165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:29.493165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:29.493165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:29.501306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:29.502756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:20:29.515114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:29.516129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:20:29.517115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:29.518281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 528/1066] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:20:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:29.522137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:29.614327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:29.631536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dd403b-9cf4-4105-bdac-a6a70955b3f6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:29.669301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:29.669899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:29.670225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:29.670501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:29.671248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:29.671553 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:29.672972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:29.696965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 539/1067] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:20:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:29.698641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:29.757361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:29.814573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee44774-3cfa-44d2-b3fc-dc0a1b33f4a8 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:29.850425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:29.851080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:29.851360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:29.851638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:29.863404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:29.864850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:29.880788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:29.881929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:29.882948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:29.882948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:29.882948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:29.887099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 529/1068] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:29.888961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:29.993032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2470b692-a3c8-49f5-bbdb-33a50b3954d7 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:30.027933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:30.028462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.028739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.029011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:30.029648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:30.029951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:30.031397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:30.048416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 540/1069] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:30.049685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:30.153454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4353530b-2d74-4bf3-9a34-08cec27f5104 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:30.160400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:30.188495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:30.189127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.189405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.189676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:30.197106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:30.198592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:30.216095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:30.217076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:30.217985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 530/1070] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:30.219874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:30.230010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:30.331438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1e0f86-d5a6-4e72-93ea-8f5dfcc0e528 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:30.368637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b95fa0-8a76-4574-856e-f934f08dd759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:30.369252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b95fa0-8a76-4574-856e-f934f08dd759 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.369525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b95fa0-8a76-4574-856e-f934f08dd759 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.369795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b95fa0-8a76-4574-856e-f934f08dd759 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:30.370498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b95fa0-8a76-4574-856e-f934f08dd759 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:30.370799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b95fa0-8a76-4574-856e-f934f08dd759 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:30.372296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b95fa0-8a76-4574-856e-f934f08dd759 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:30.386307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 541/1071] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:20:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:30.392138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:30.467702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc70f38c-408a-4c43-904e-ed8ca942adf1 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.468051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc70f38c-408a-4c43-904e-ed8ca942adf1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.468265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc70f38c-408a-4c43-904e-ed8ca942adf1 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:30.469596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 542/1072] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:20:30.474621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb77914-7472-41f5-a69e-7197bca207df None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.474896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb77914-7472-41f5-a69e-7197bca207df None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.475197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb77914-7472-41f5-a69e-7197bca207df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:30.494672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0bb77914-7472-41f5-a69e-7197bca207df None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:30.496310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0bb77914-7472-41f5-a69e-7197bca207df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:30.499758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b54d0-dc40-4bb8-86e4-8814ec0a934f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:30.535153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:30.535811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.536088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.537018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/c8d2db883de44964b6e15b85a97726b5/roles/f1d374a285264fe38d7173e7d7603f55` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:30.538893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=c8d2db883de44964b6e15b85a97726b5, role_id=f1d374a285264fe38d7173e7d7603f55)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:30.540575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:30.563004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 531/1073] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:20:30 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/c8d2db883de44964b6e15b85a97726b5/roles/f1d374a285264fe38d7173e7d7603f55 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:20:30.570992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:30.581605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 543/1074] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:30.852850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:30.886266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:30.964834 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d601ef-af3a-419e-9c84-6dfdac6fa6c9 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:30.983731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df342aa0-9116-4eda-be96-5f0cd20f0a9d None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.984274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df342aa0-9116-4eda-be96-5f0cd20f0a9d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.984274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df342aa0-9116-4eda-be96-5f0cd20f0a9d None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:30.985806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 544/1075] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:30.990907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb2e25-01e4-4672-a8ef-67cea3fc79d1 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:30.991215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb2e25-01e4-4672-a8ef-67cea3fc79d1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:30.991487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacb2e25-01e4-4672-a8ef-67cea3fc79d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.000917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:31.001549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:31.001826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:31.002097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.012428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-dacb2e25-01e4-4672-a8ef-67cea3fc79d1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:31.013622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dacb2e25-01e4-4672-a8ef-67cea3fc79d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:31.013997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:31.019755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:31.020722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:31.021589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 532/1076] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:20:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:31.027357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:31.095013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 545/1077] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:31.128992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:31.145260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a776f96-9912-45f1-8143-7c6a36e387fe None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:31.180528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:31.181168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:31.181450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:31.181725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.182382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:31.182684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:31.184075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:31.197745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 533/1078] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:20:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:31.199383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb2e25-01e4-4672-a8ef-67cea3fc79d1 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:31.307152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacb2e25-01e4-4672-a8ef-67cea3fc79d1 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:31.330375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:31.342194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:31.342823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:31.343171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:31.343449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] PATH_INFO: `/v3/users/3bcfea5e66e7464691dfa58e11a8c485` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.344565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] RBAC: Authorizing `identity:get_user(user_id=3bcfea5e66e7464691dfa58e11a8c485)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:31.345991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:31.352671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 546/1079] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:31 2022] GET /identity/v3/users/3bcfea5e66e7464691dfa58e11a8c485 => generated 285 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:31.353875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:31.378275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:31.459679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ad909-206e-4e4b-8f9e-7ef008c8ceab None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:31.487321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d731c-f62f-468e-b0a6-4a29fcb48cad None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:31.501016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:31.501953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:31.502431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:31.502912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.504264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] RBAC: Authorizing `identity:get_project(project_id=84ce0d06721640798467c802ae991b78)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:31.505900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:31.507889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 534/1080] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:20:31 2022] GET /identity/v3/projects/84ce0d06721640798467c802ae991b78 => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:31.514377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:31.522761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:31.523397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:31.523730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:31.524010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.536330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:31.583878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:31.596426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:31.597875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:31.628381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c72ef-c769-4b88-a694-ebf4ea9d4c5b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:31.665614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:31.666308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:31.666587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:31.666862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78/users/3bcfea5e66e7464691dfa58e11a8c485/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.668818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] RBAC: Authorizing `identity:create_grant(project_id=84ce0d06721640798467c802ae991b78, user_id=3bcfea5e66e7464691dfa58e11a8c485, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:31.670242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:31.684760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 535/1081] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:20:31 2022] PUT /identity/v3/projects/84ce0d06721640798467c802ae991b78/users/3bcfea5e66e7464691dfa58e11a8c485/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:20:31.706994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 547/1082] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 06 04:20:31.713366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:31.804161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:31.833310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c5e4b0-76de-47fb-8463-0c63b24e90ea None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:31.874387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0306dcc6-10be-42db-a209-f093969fe8e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:31.875103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0306dcc6-10be-42db-a209-f093969fe8e1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:31.875328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0306dcc6-10be-42db-a209-f093969fe8e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:31.875583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0306dcc6-10be-42db-a209-f093969fe8e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:31.888195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0306dcc6-10be-42db-a209-f093969fe8e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:31.889842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0306dcc6-10be-42db-a209-f093969fe8e1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-0306dcc6-10be-42db-a209-f093969fe8e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:31.904723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:31.905699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:31.907257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:31.907257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:31.907257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:31.907257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 536/1083] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:31.911727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:32.004080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:32.014600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517ecff7-89ff-44e4-8be2-d66d470987bc None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:32.051811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a71f52-8e93-4d38-8604-6940b3f01172 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:32.052404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a71f52-8e93-4d38-8604-6940b3f01172 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:32.052671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a71f52-8e93-4d38-8604-6940b3f01172 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:32.052941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a71f52-8e93-4d38-8604-6940b3f01172 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:32.053577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a71f52-8e93-4d38-8604-6940b3f01172 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:32.053875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a71f52-8e93-4d38-8604-6940b3f01172 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:32.055826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a71f52-8e93-4d38-8604-6940b3f01172 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:32.067213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 548/1084] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:32.717349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233eb390-8e18-4425-8733-807dd058fcc1 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:32.717639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233eb390-8e18-4425-8733-807dd058fcc1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:32.717917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233eb390-8e18-4425-8733-807dd058fcc1 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:32.719433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 537/1085] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:32.724836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723589f8-aab9-4d16-8510-d36e646318b9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:32.725125 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723589f8-aab9-4d16-8510-d36e646318b9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:32.725402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723589f8-aab9-4d16-8510-d36e646318b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:32.744151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-723589f8-aab9-4d16-8510-d36e646318b9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:32.745327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-723589f8-aab9-4d16-8510-d36e646318b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:32.809633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 549/1086] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:33.085722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233eb390-8e18-4425-8733-807dd058fcc1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:33.186439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233eb390-8e18-4425-8733-807dd058fcc1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:33.211576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:33.220264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:33.220888 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:33.221164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:33.221433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:33.232565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:33.237569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:33.238539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 06 04:20:33.239432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:33.243372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 538/1087] 10.222.0.33 () {56 vars in 1120 bytes} [Thu Oct 6 04:20:33 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:33.245123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723589f8-aab9-4d16-8510-d36e646318b9 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:33.323606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:33.342311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723589f8-aab9-4d16-8510-d36e646318b9 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:33.376727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:33.377320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:33.377593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:33.377861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:33.378530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:33.378829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:33.380201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:33.392808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 550/1088] 10.222.0.33 () {56 vars in 1115 bytes} [Thu Oct 6 04:20:33 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:33.394190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:33.494512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb078f40-dc43-4483-8582-fd2586c8a162 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:33.527745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:33.528368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:33.528642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:33.528911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:33.539226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:33.541161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:33.554671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:33.555761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:33.556692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:33.556692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:33.556692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:33.556692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 539/1089] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:33.561627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:33.657850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ff1108-d591-4065-b8a2-0f0c0f06bd3b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:33.689740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec98bc5-cc3d-444e-be6c-9787eb0dbba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:33.690322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec98bc5-cc3d-444e-be6c-9787eb0dbba0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:33.690592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec98bc5-cc3d-444e-be6c-9787eb0dbba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:33.690856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec98bc5-cc3d-444e-be6c-9787eb0dbba0 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:33.691520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec98bc5-cc3d-444e-be6c-9787eb0dbba0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:33.691799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec98bc5-cc3d-444e-be6c-9787eb0dbba0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:33.693129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec98bc5-cc3d-444e-be6c-9787eb0dbba0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:33.705036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 551/1090] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:33.710595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:33.752718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:33.813095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd0e0b-8773-4add-80af-cad62cb155ce None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:33.846314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:33.846930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:33.847227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:33.847494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:33.855507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:33.856938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:20:33.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:33.870090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:33.871304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 540/1091] 10.222.0.33 () {56 vars in 1093 bytes} [Thu Oct 6 04:20:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:33.872671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894fecf4-fb0e-4be7-90ac-1364e9083c28 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:33.872957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894fecf4-fb0e-4be7-90ac-1364e9083c28 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:33.873232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894fecf4-fb0e-4be7-90ac-1364e9083c28 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:33.874849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 552/1092] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:33.875295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:33.880166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281c60ad-11e3-4087-819a-c710ffe383b0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:33.880438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281c60ad-11e3-4087-819a-c710ffe383b0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:33.880705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281c60ad-11e3-4087-819a-c710ffe383b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:33.887897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:33.899224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-281c60ad-11e3-4087-819a-c710ffe383b0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:33.900466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-281c60ad-11e3-4087-819a-c710ffe383b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:33.969190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 553/1093] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:33.972523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:33.975539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d2b41-1ab5-45bc-862b-079a6d852d83 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.009643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.010237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.010514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.010783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.011529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:34.011831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.013246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.029900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 541/1094] 10.222.0.33 () {56 vars in 1133 bytes} [Thu Oct 6 04:20:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:34.035751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281c60ad-11e3-4087-819a-c710ffe383b0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.135976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281c60ad-11e3-4087-819a-c710ffe383b0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.169307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.169901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.170177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.170447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.172972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:34.236617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.240248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.241686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.335409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16fcc56-903d-4e87-91cf-098689a9b741 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.338309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 554/1095] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:20:34.344622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.370875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.371531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.371810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.372089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.378096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:34.382150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:34.387297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:34.388434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:34.389503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 542/1096] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:34.394240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.413582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:34.447057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddb590e-02fa-4aaf-9456-dbbcce31dcf6 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.481455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.482076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.482354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.482625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.493727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.495124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.497121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0774a-2a34-4c3a-b45e-cdffc668e76b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:20:34.509947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:34.510994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:20:34.512278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:20:34.512278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:20:34.512278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:34.512278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 555/1097] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:34.516596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.535597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.536251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.536491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.536753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.537395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:34.537694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.539108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.547221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 543/1098] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:34.552952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.602884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:34.618891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0884f0d-88cf-4016-b418-ae73abd91dd5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.632827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:34.652188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8368e02-1bae-43dc-b8ff-7ed01bfc94f7 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.653131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.653728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.654004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.654273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.654910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:34.655240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.656591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.668592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 556/1099] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:34.674051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.686632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.687288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.687567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.687836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] PATH_INFO: `/v3/users/3bcfea5e66e7464691dfa58e11a8c485` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.688952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] RBAC: Authorizing `identity:get_user(user_id=3bcfea5e66e7464691dfa58e11a8c485)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.690381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.697413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 544/1100] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/users/3bcfea5e66e7464691dfa58e11a8c485 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:34.698649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.773208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f309fdc-c7fb-45f3-bc43-93eece329e75 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.799543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dc94ed-1532-4a6d-a36b-14f34c6c9099 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.813029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.813898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.814338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.814787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.822932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.824456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:20:34.835419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:34.837088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:20:34.838130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 557/1101] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:20:34.840185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.840787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.841064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.841338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.842868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:34.850739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:34.859606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:34.910727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.912202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.941617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe91473-69a4-4830-9adb-6432c25ed8ce None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:34.974850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:34.975471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:34.975746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:34.976013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:34.976741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:34.977338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:34.978527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:34.990389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 558/1102] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:34.997011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:35.012872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 545/1103] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:20:35.018586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:35.063787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:35.066291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:35.101319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a53d68-bbfe-4445-8c62-1890bd8469d2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:35.120703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e7a0c-427a-417b-bbca-49d88a1a546a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:35.135361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:35.135946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:35.136217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:35.136482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:35.137187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] RBAC: Adding query filter params (role.id=f1d374a285264fe38d7173e7d7603f55, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=c8d2db883de44964b6e15b85a97726b5) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:35.137488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:35.138893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:35.144484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 559/1104] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:34 2022] GET /identity/v3/role_assignments?user.id=c8d2db883de44964b6e15b85a97726b5&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=f1d374a285264fe38d7173e7d7603f55 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:35.157792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce99fcc9-b96e-4b5b-a1d2-5e82055f3edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:35.158442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99fcc9-b96e-4b5b-a1d2-5e82055f3edb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:35.158728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99fcc9-b96e-4b5b-a1d2-5e82055f3edb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:35.159043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99fcc9-b96e-4b5b-a1d2-5e82055f3edb None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:35.160234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce99fcc9-b96e-4b5b-a1d2-5e82055f3edb None admin] RBAC: Authorizing `identity:get_project(project_id=84ce0d06721640798467c802ae991b78)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:35.161720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce99fcc9-b96e-4b5b-a1d2-5e82055f3edb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:35.163507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 546/1105] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:20:35 2022] GET /identity/v3/projects/84ce0d06721640798467c802ae991b78 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:35.170452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:35.275908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd714197-a965-447f-9eed-1faf22287bc0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:35.305758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:35.312458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b6c3ab-a5bf-4614-8247-ec5247c24a50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:35.313055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6c3ab-a5bf-4614-8247-ec5247c24a50 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:35.313326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6c3ab-a5bf-4614-8247-ec5247c24a50 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:35.313601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b6c3ab-a5bf-4614-8247-ec5247c24a50 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:35.314372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b6c3ab-a5bf-4614-8247-ec5247c24a50 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=84ce0d06721640798467c802ae991b78, user.id=3bcfea5e66e7464691dfa58e11a8c485) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:35.314617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b6c3ab-a5bf-4614-8247-ec5247c24a50 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:35.316062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b6c3ab-a5bf-4614-8247-ec5247c24a50 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:35.322594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 560/1106] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:35 2022] GET /identity/v3/role_assignments?user.id=3bcfea5e66e7464691dfa58e11a8c485&scope.project.id=84ce0d06721640798467c802ae991b78&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:37.471829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8663a7e1-24d2-4eb4-84dc-e73f58442d77 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:37.472349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8663a7e1-24d2-4eb4-84dc-e73f58442d77 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:37.472349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8663a7e1-24d2-4eb4-84dc-e73f58442d77 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:37.473599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 547/1107] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:37.478823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf15be-ff35-4a84-81ad-501da9a998fd None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:37.479278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf15be-ff35-4a84-81ad-501da9a998fd None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:37.479480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf15be-ff35-4a84-81ad-501da9a998fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:37.498066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0abf15be-ff35-4a84-81ad-501da9a998fd None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:37.499274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0abf15be-ff35-4a84-81ad-501da9a998fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:37.563442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 561/1108] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:37.831476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8663a7e1-24d2-4eb4-84dc-e73f58442d77 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:37.934296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8663a7e1-24d2-4eb4-84dc-e73f58442d77 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:37.950630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:37.968706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444c5ffa-36c4-491f-9cd9-3faab607d4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:37.969321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444c5ffa-36c4-491f-9cd9-3faab607d4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:37.969595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444c5ffa-36c4-491f-9cd9-3faab607d4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:37.969864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444c5ffa-36c4-491f-9cd9-3faab607d4c6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:37.970985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444c5ffa-36c4-491f-9cd9-3faab607d4c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:37.972414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444c5ffa-36c4-491f-9cd9-3faab607d4c6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:37.973958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 548/1109] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:37.979840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf15be-ff35-4a84-81ad-501da9a998fd None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:38.073556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf15be-ff35-4a84-81ad-501da9a998fd None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:38.096253 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:38.106808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495e3955-4adf-4d9a-8ede-9ddcc6acf8c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:38.107413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495e3955-4adf-4d9a-8ede-9ddcc6acf8c0 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:38.107713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495e3955-4adf-4d9a-8ede-9ddcc6acf8c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:38.107977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495e3955-4adf-4d9a-8ede-9ddcc6acf8c0 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:38.108618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495e3955-4adf-4d9a-8ede-9ddcc6acf8c0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:38.110020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495e3955-4adf-4d9a-8ede-9ddcc6acf8c0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:38.130723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 562/1110] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:20:37 2022] POST /identity/v3/users => generated 286 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:40.284992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad865566-5b79-488f-8ba9-5b28f955ca5b None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:40.285518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad865566-5b79-488f-8ba9-5b28f955ca5b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:40.285518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad865566-5b79-488f-8ba9-5b28f955ca5b None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:40.286844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 549/1111] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:40.292668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ebf1d-3859-4013-a437-7f76c53fa9a3 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:40.293107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ebf1d-3859-4013-a437-7f76c53fa9a3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:40.293210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ebf1d-3859-4013-a437-7f76c53fa9a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:40.311295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-fd2ebf1d-3859-4013-a437-7f76c53fa9a3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:40.312470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fd2ebf1d-3859-4013-a437-7f76c53fa9a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:40.377513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 563/1112] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:40.646431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad865566-5b79-488f-8ba9-5b28f955ca5b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:40.745601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad865566-5b79-488f-8ba9-5b28f955ca5b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:40.778763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:40.779398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:40.779674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:40.779944 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] PATH_INFO: `/v3/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:40.781307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] RBAC: Authorizing `identity:get_role(role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:40.782659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:40.783990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 550/1113] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:40 2022] GET /identity/v3/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:40.789861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ebf1d-3859-4013-a437-7f76c53fa9a3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:40.885709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ebf1d-3859-4013-a437-7f76c53fa9a3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:40.896946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:40.926346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:40.927301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:40.927715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:40.928122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] PATH_INFO: `/v3/users/7421c6f44a344ad2880aa4fb72050dd7` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:40.947976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] RBAC: Authorizing `identity:get_user(user_id=7421c6f44a344ad2880aa4fb72050dd7)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:40.950229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:40.952758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 564/1114] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:40 2022] GET /identity/v3/users/7421c6f44a344ad2880aa4fb72050dd7 => generated 286 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:40.958523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:41.010422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:41.055351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206a202e-2ae3-4740-9c3b-4d5ffebba466 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:41.088556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:41.089190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:41.089413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:41.090432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:41.155963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:41.157360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:41.206398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:41.253446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 551/1115] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:20:41.259927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:41.363259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818676bc-6efd-498f-924d-da3a283410d5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:41.398000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d849cfac-60dc-49a4-8da1-18f7b9ae3190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:41.398763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d849cfac-60dc-49a4-8da1-18f7b9ae3190 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:41.399082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d849cfac-60dc-49a4-8da1-18f7b9ae3190 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:41.399357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d849cfac-60dc-49a4-8da1-18f7b9ae3190 None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:41.400457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d849cfac-60dc-49a4-8da1-18f7b9ae3190 None admin] RBAC: Authorizing `identity:get_project(project_id=84ce0d06721640798467c802ae991b78)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:41.401861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d849cfac-60dc-49a4-8da1-18f7b9ae3190 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:41.403560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 565/1116] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:20:41 2022] GET /identity/v3/projects/84ce0d06721640798467c802ae991b78 => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:41.409753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:41.506287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81a7462-bd53-4de3-bc03-c3ead2df6c6c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:41.538802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f648526-c9bf-4769-a53e-01b5e384e69d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:41.540087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f648526-c9bf-4769-a53e-01b5e384e69d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:41.540358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f648526-c9bf-4769-a53e-01b5e384e69d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:41.540626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f648526-c9bf-4769-a53e-01b5e384e69d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:41.541336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f648526-c9bf-4769-a53e-01b5e384e69d None admin] RBAC: Adding query filter params (role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=84ce0d06721640798467c802ae991b78, user.id=7421c6f44a344ad2880aa4fb72050dd7) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:41.541639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f648526-c9bf-4769-a53e-01b5e384e69d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:41.543081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f648526-c9bf-4769-a53e-01b5e384e69d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:41.548239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 552/1117] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:41 2022] GET /identity/v3/role_assignments?user.id=7421c6f44a344ad2880aa4fb72050dd7&scope.project.id=84ce0d06721640798467c802ae991b78&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:43.655214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3e0af-2d1e-4896-979f-0049e98bd286 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:43.655781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3e0af-2d1e-4896-979f-0049e98bd286 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:43.655781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3e0af-2d1e-4896-979f-0049e98bd286 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:43.657109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 566/1118] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:43.662063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa85388-d48e-4e24-89fd-f26e3ee81dad None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:43.662468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa85388-d48e-4e24-89fd-f26e3ee81dad None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:43.662603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa85388-d48e-4e24-89fd-f26e3ee81dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:43.681469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dfa85388-d48e-4e24-89fd-f26e3ee81dad None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:43.682645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dfa85388-d48e-4e24-89fd-f26e3ee81dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:43.692999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:43.747545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 553/1119] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:44.011081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3e0af-2d1e-4896-979f-0049e98bd286 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:44.046547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:44.109586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3e0af-2d1e-4896-979f-0049e98bd286 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:44.142671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:44.143330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:44.143609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:44.143940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] PATH_INFO: `/v3/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:44.145290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] RBAC: Authorizing `identity:get_role(role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:44.146622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:44.147936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 567/1120] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:44 2022] GET /identity/v3/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:44.153806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa85388-d48e-4e24-89fd-f26e3ee81dad None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:44.251087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa85388-d48e-4e24-89fd-f26e3ee81dad None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:44.284898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:44.285452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:44.285729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:44.286000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] PATH_INFO: `/v3/users/7421c6f44a344ad2880aa4fb72050dd7` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:44.287125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] RBAC: Authorizing `identity:get_user(user_id=7421c6f44a344ad2880aa4fb72050dd7)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:44.288545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:44.290302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 554/1121] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:44 2022] GET /identity/v3/users/7421c6f44a344ad2880aa4fb72050dd7 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:44.296053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:44.391328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdfeac5-0ac7-4990-81c6-ba3477161d89 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:44.393531 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:44.424327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49d32f0-adc9-4600-bf4d-4f110f93a11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:44.424931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49d32f0-adc9-4600-bf4d-4f110f93a11e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:44.425203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49d32f0-adc9-4600-bf4d-4f110f93a11e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:44.425467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49d32f0-adc9-4600-bf4d-4f110f93a11e None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:44.426567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49d32f0-adc9-4600-bf4d-4f110f93a11e None admin] RBAC: Authorizing `identity:get_project(project_id=84ce0d06721640798467c802ae991b78)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:44.427999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49d32f0-adc9-4600-bf4d-4f110f93a11e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:44.430180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 568/1122] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:20:44 2022] GET /identity/v3/projects/84ce0d06721640798467c802ae991b78 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:44.435996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:44.451220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:44.542349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f1c9a2-391a-4826-97d3-ef403cb9604c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:44.575829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4f7857-bf7b-4144-81ac-862351f96294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:44.576475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4f7857-bf7b-4144-81ac-862351f96294 None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:44.576757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4f7857-bf7b-4144-81ac-862351f96294 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:44.577027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4f7857-bf7b-4144-81ac-862351f96294 None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78/users/7421c6f44a344ad2880aa4fb72050dd7/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:44.579201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4f7857-bf7b-4144-81ac-862351f96294 None admin] RBAC: Authorizing `identity:create_grant(project_id=84ce0d06721640798467c802ae991b78, user_id=7421c6f44a344ad2880aa4fb72050dd7, role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:44.580634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4f7857-bf7b-4144-81ac-862351f96294 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:44.592825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 555/1123] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:20:44 2022] PUT /identity/v3/projects/84ce0d06721640798467c802ae991b78/users/7421c6f44a344ad2880aa4fb72050dd7/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:20:46.794679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540eeac-807f-4907-b81d-c6ee0d56e775 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:46.795248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540eeac-807f-4907-b81d-c6ee0d56e775 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:46.795248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540eeac-807f-4907-b81d-c6ee0d56e775 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:46.796591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 569/1124] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:46.801883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84212e-13a5-401b-b96e-1d1cb7febef1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:46.802310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84212e-13a5-401b-b96e-1d1cb7febef1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:46.802412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84212e-13a5-401b-b96e-1d1cb7febef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:46.820838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fc84212e-13a5-401b-b96e-1d1cb7febef1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:46.822000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fc84212e-13a5-401b-b96e-1d1cb7febef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:46.885522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 556/1125] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:47.163674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9540eeac-807f-4907-b81d-c6ee0d56e775 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:47.263710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9540eeac-807f-4907-b81d-c6ee0d56e775 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:47.295857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:47.296463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:47.296736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:47.297004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] PATH_INFO: `/v3/roles/07adffdee0794f21b7708be9f5ee8f2b` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:47.298369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] RBAC: Authorizing `identity:get_role(role_id=07adffdee0794f21b7708be9f5ee8f2b)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:47.299746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:47.301042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 570/1126] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:47 2022] GET /identity/v3/roles/07adffdee0794f21b7708be9f5ee8f2b => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:47.307476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84212e-13a5-401b-b96e-1d1cb7febef1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:47.310508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:47.404919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84212e-13a5-401b-b96e-1d1cb7febef1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:47.437307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:47.437858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:47.438135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:47.438405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] PATH_INFO: `/v3/users/7421c6f44a344ad2880aa4fb72050dd7` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:47.439520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] RBAC: Authorizing `identity:get_user(user_id=7421c6f44a344ad2880aa4fb72050dd7)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:47.440947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:47.442490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 557/1127] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:47 2022] GET /identity/v3/users/7421c6f44a344ad2880aa4fb72050dd7 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:47.448921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:47.489924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:47.546066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074c5ec-9241-426d-87c4-f4303f2de436 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:47.578725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:47.579334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:47.579611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:47.579878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:47.643943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:47.645622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:47.685425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:47.742820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 571/1128] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:47 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:20:47.748670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:47.816414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:47.845415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4c9868-421a-4ed1-97ee-95883780bcfb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:47.878595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a78bfb6-7753-4183-8d16-f2cfe6ab3f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:47.879230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a78bfb6-7753-4183-8d16-f2cfe6ab3f46 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:47.879504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a78bfb6-7753-4183-8d16-f2cfe6ab3f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:47.879772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a78bfb6-7753-4183-8d16-f2cfe6ab3f46 None admin] PATH_INFO: `/v3/projects/84ce0d06721640798467c802ae991b78` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:47.880858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a78bfb6-7753-4183-8d16-f2cfe6ab3f46 None admin] RBAC: Authorizing `identity:get_project(project_id=84ce0d06721640798467c802ae991b78)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:47.882297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a78bfb6-7753-4183-8d16-f2cfe6ab3f46 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:47.884573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 558/1129] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:20:47 2022] GET /identity/v3/projects/84ce0d06721640798467c802ae991b78 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:47.890642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:47.985094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec51183-554d-46ad-a238-cdd0446faa4f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:48.017819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3db26e-ac60-4ce4-a647-2a011cbe872b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:48.018337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3db26e-ac60-4ce4-a647-2a011cbe872b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:48.018607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3db26e-ac60-4ce4-a647-2a011cbe872b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:48.018874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3db26e-ac60-4ce4-a647-2a011cbe872b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:48.019603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3db26e-ac60-4ce4-a647-2a011cbe872b None admin] RBAC: Adding query filter params (role.id=07adffdee0794f21b7708be9f5ee8f2b, scope.project.id=84ce0d06721640798467c802ae991b78, user.id=7421c6f44a344ad2880aa4fb72050dd7) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:48.019903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3db26e-ac60-4ce4-a647-2a011cbe872b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:48.021289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3db26e-ac60-4ce4-a647-2a011cbe872b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:48.027339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 572/1130] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:47 2022] GET /identity/v3/role_assignments?user.id=7421c6f44a344ad2880aa4fb72050dd7&scope.project.id=84ce0d06721640798467c802ae991b78&role.id=07adffdee0794f21b7708be9f5ee8f2b => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:50.258823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd7eaa-77e2-469d-b61d-aebfac3ba5d8 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:50.259410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd7eaa-77e2-469d-b61d-aebfac3ba5d8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:50.259410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd7eaa-77e2-469d-b61d-aebfac3ba5d8 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:50.260720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 559/1131] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:50.266097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faff4c9-3f3f-4460-ae76-c27837f6c16a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:50.266522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faff4c9-3f3f-4460-ae76-c27837f6c16a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:50.266631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faff4c9-3f3f-4460-ae76-c27837f6c16a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:50.275492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:50.286981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7faff4c9-3f3f-4460-ae76-c27837f6c16a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:50.288159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7faff4c9-3f3f-4460-ae76-c27837f6c16a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:50.356799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 573/1132] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:50.634733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd7eaa-77e2-469d-b61d-aebfac3ba5d8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:50.739452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd7eaa-77e2-469d-b61d-aebfac3ba5d8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:50.774459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d9a523-e271-41a0-8e99-c849eb6ffd06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:50.775118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d9a523-e271-41a0-8e99-c849eb6ffd06 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:50.775396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d9a523-e271-41a0-8e99-c849eb6ffd06 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:50.775664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d9a523-e271-41a0-8e99-c849eb6ffd06 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:50.776777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d9a523-e271-41a0-8e99-c849eb6ffd06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:50.778209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d9a523-e271-41a0-8e99-c849eb6ffd06 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:50.779866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 560/1133] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:50.785669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faff4c9-3f3f-4460-ae76-c27837f6c16a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:50.884860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faff4c9-3f3f-4460-ae76-c27837f6c16a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:50.920685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca43e6f-76e7-45f2-8856-dfd438c0b59c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:50.921278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca43e6f-76e7-45f2-8856-dfd438c0b59c None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:50.921551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca43e6f-76e7-45f2-8856-dfd438c0b59c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:50.921816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca43e6f-76e7-45f2-8856-dfd438c0b59c None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:50.922452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca43e6f-76e7-45f2-8856-dfd438c0b59c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:50.923893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca43e6f-76e7-45f2-8856-dfd438c0b59c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:50.928288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:50.949659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 574/1134] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:20:50 2022] POST /identity/v3/projects => generated 311 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:53.099976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f537cd2e-2593-46a5-b011-ddb36784ec2a None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:53.100517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f537cd2e-2593-46a5-b011-ddb36784ec2a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:53.100517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f537cd2e-2593-46a5-b011-ddb36784ec2a None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:53.101849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 561/1135] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:53.108814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b203d1c3-e144-4998-938b-f1773130da88 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:53.109484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b203d1c3-e144-4998-938b-f1773130da88 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:53.109780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b203d1c3-e144-4998-938b-f1773130da88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:53.128729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b203d1c3-e144-4998-938b-f1773130da88 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:53.129916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b203d1c3-e144-4998-938b-f1773130da88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:53.196126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 575/1136] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:20:53.469831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f537cd2e-2593-46a5-b011-ddb36784ec2a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:53.481210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:53.569332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f537cd2e-2593-46a5-b011-ddb36784ec2a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:53.603697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9aaadf1-1cff-4445-963a-1119b2d508c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:53.604304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aaadf1-1cff-4445-963a-1119b2d508c7 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:53.604577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aaadf1-1cff-4445-963a-1119b2d508c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:53.604845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aaadf1-1cff-4445-963a-1119b2d508c7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:53.605922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9aaadf1-1cff-4445-963a-1119b2d508c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:53.607463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9aaadf1-1cff-4445-963a-1119b2d508c7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:53.608993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 562/1137] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:20:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:53.614924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b203d1c3-e144-4998-938b-f1773130da88 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:53.714017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b203d1c3-e144-4998-938b-f1773130da88 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:53.730691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:53.749128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a80c7-eb84-4c2f-9746-d80bed330b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:53.749715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a80c7-eb84-4c2f-9746-d80bed330b4c None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:53.749992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a80c7-eb84-4c2f-9746-d80bed330b4c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:53.750263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a80c7-eb84-4c2f-9746-d80bed330b4c None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:53.750907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a80c7-eb84-4c2f-9746-d80bed330b4c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:53.752347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a80c7-eb84-4c2f-9746-d80bed330b4c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:53.772443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 576/1138] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:20:53 2022] POST /identity/v3/users => generated 286 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:20:55.981020 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faaf5a5-56f4-4ff7-98aa-db4f5a98c3d9 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:55.981550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faaf5a5-56f4-4ff7-98aa-db4f5a98c3d9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:55.981550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faaf5a5-56f4-4ff7-98aa-db4f5a98c3d9 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:55.982878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 563/1139] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:55.988418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f21bc6b-a904-46d4-b8ae-31297ceea5ca None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:55.988797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f21bc6b-a904-46d4-b8ae-31297ceea5ca None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:55.988960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f21bc6b-a904-46d4-b8ae-31297ceea5ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:56.006608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2f21bc6b-a904-46d4-b8ae-31297ceea5ca None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:56.007855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2f21bc6b-a904-46d4-b8ae-31297ceea5ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:56.071758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 577/1140] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:56.342659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faaf5a5-56f4-4ff7-98aa-db4f5a98c3d9 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:56.418621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:56.442326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faaf5a5-56f4-4ff7-98aa-db4f5a98c3d9 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:56.475117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:56.476409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:56.476690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:56.476961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:56.486899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:56.491734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:56.492880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:56.493724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 564/1141] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:20:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:56.498713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f21bc6b-a904-46d4-b8ae-31297ceea5ca None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:56.597107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f21bc6b-a904-46d4-b8ae-31297ceea5ca None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:56.598513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:56.632493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:56.633083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:56.633355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:56.633620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:56.634267 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:56.634564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:56.635935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:56.643579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 578/1142] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:20:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:56.650256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:56.746909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a602e8b1-8595-4af3-9faf-5cc88f5fa874 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:56.779465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:56.780072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:56.780349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:56.780622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] PATH_INFO: `/v3/users/3c24c091db5f474083170fe78dd35b79` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:56.783210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:56.793629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] RBAC: Authorizing `identity:get_user(user_id=3c24c091db5f474083170fe78dd35b79)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:56.795084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:56.796658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 565/1143] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:20:56 2022] GET /identity/v3/users/3c24c091db5f474083170fe78dd35b79 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:56.802453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:56.902062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43143243-a0d7-42a6-bebe-b03751cd56c2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:56.935011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:56.935603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:56.935873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:56.936140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:56.965828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:57.002393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:57.003850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:57.097177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 579/1144] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:20:56 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:20:57.103785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:57.202421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb0fb5c-59f2-495c-9228-ed70072c5352 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:57.235816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b661536-1d47-4d06-ab12-c08488b088bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:57.236714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b661536-1d47-4d06-ab12-c08488b088bb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:57.236986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b661536-1d47-4d06-ab12-c08488b088bb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:57.237253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b661536-1d47-4d06-ab12-c08488b088bb None admin] PATH_INFO: `/v3/projects/7656a1b184e046de88cdfcd4f391657e` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:57.238389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b661536-1d47-4d06-ab12-c08488b088bb None admin] RBAC: Authorizing `identity:get_project(project_id=7656a1b184e046de88cdfcd4f391657e)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:57.239873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b661536-1d47-4d06-ab12-c08488b088bb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:57.242518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 566/1145] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:20:57 2022] GET /identity/v3/projects/7656a1b184e046de88cdfcd4f391657e => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:20:57.248959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:57.310839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:57.346508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96edfc5-884e-4ed3-9529-5c9cc311830f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:57.380010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ca3b2a-a28a-4c3d-a4ce-a36fee607f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:57.380620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ca3b2a-a28a-4c3d-a4ce-a36fee607f35 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:57.380894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ca3b2a-a28a-4c3d-a4ce-a36fee607f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:57.381161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ca3b2a-a28a-4c3d-a4ce-a36fee607f35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:57.381891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ca3b2a-a28a-4c3d-a4ce-a36fee607f35 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=7656a1b184e046de88cdfcd4f391657e, user.id=3c24c091db5f474083170fe78dd35b79) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:20:57.382222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ca3b2a-a28a-4c3d-a4ce-a36fee607f35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:20:57.383756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ca3b2a-a28a-4c3d-a4ce-a36fee607f35 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:20:57.388920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 580/1146] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:20:57 2022] GET /identity/v3/role_assignments?user.id=3c24c091db5f474083170fe78dd35b79&scope.project.id=7656a1b184e046de88cdfcd4f391657e&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:20:59.472908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cddf8fd-ab26-4930-a928-aededf0ef49f None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:59.473409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cddf8fd-ab26-4930-a928-aededf0ef49f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:59.473409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cddf8fd-ab26-4930-a928-aededf0ef49f None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:59.474718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 567/1147] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:20:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:20:59.480562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2eb195b-8924-4088-adbc-1d8e04d74473 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:59.480981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2eb195b-8924-4088-adbc-1d8e04d74473 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:59.481087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2eb195b-8924-4088-adbc-1d8e04d74473 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:59.499801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a2eb195b-8924-4088-adbc-1d8e04d74473 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:20:59.500980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a2eb195b-8924-4088-adbc-1d8e04d74473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:20:59.564946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 581/1148] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:20:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:20:59.826907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cddf8fd-ab26-4930-a928-aededf0ef49f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:20:59.888419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:20:59.927928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cddf8fd-ab26-4930-a928-aededf0ef49f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:20:59.961675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:20:59.962283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:20:59.962558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:20:59.962825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:20:59.972579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:20:59.977890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:20:59.979014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:20:59.980082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 568/1149] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:20:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:20:59.985094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2eb195b-8924-4088-adbc-1d8e04d74473 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:00.040935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:00.081543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2eb195b-8924-4088-adbc-1d8e04d74473 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:00.119420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:00.120272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:00.120680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:00.121080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:00.122065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:00.122517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:00.124630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:00.137512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 582/1150] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:20:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:00.143714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:00.243244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a875f95e-2a46-49c3-859a-91c650b130c4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:00.265266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:00.277457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:00.278070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:00.278347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:00.278617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] PATH_INFO: `/v3/users/3c24c091db5f474083170fe78dd35b79` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:00.279739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] RBAC: Authorizing `identity:get_user(user_id=3c24c091db5f474083170fe78dd35b79)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:00.281155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:00.282696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 569/1151] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:21:00 2022] GET /identity/v3/users/3c24c091db5f474083170fe78dd35b79 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:00.288684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:00.384029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209b7173-49a3-4d30-8a4c-8a0621de51ad None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:00.402374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:00.416713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ca2807-adb7-4cd3-bdbb-8b4a1224d719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:00.417317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ca2807-adb7-4cd3-bdbb-8b4a1224d719 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:00.417590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ca2807-adb7-4cd3-bdbb-8b4a1224d719 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:00.417857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ca2807-adb7-4cd3-bdbb-8b4a1224d719 None admin] PATH_INFO: `/v3/projects/7656a1b184e046de88cdfcd4f391657e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:00.418938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ca2807-adb7-4cd3-bdbb-8b4a1224d719 None admin] RBAC: Authorizing `identity:get_project(project_id=7656a1b184e046de88cdfcd4f391657e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:00.420407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ca2807-adb7-4cd3-bdbb-8b4a1224d719 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:00.421998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 583/1152] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:21:00 2022] GET /identity/v3/projects/7656a1b184e046de88cdfcd4f391657e => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:00.427844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:00.531873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73965585-d484-4182-ae66-090f8cee1cc3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:00.564620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f0a5a4-f925-47e8-acef-c9ec88d07f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:00.565257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f0a5a4-f925-47e8-acef-c9ec88d07f4c None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:00.565529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f0a5a4-f925-47e8-acef-c9ec88d07f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:00.565802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f0a5a4-f925-47e8-acef-c9ec88d07f4c None admin] PATH_INFO: `/v3/projects/7656a1b184e046de88cdfcd4f391657e/users/3c24c091db5f474083170fe78dd35b79/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:00.567652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f0a5a4-f925-47e8-acef-c9ec88d07f4c None admin] RBAC: Authorizing `identity:create_grant(project_id=7656a1b184e046de88cdfcd4f391657e, user_id=3c24c091db5f474083170fe78dd35b79, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:00.569066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f0a5a4-f925-47e8-acef-c9ec88d07f4c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:00.582730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 570/1153] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:21:00 2022] PUT /identity/v3/projects/7656a1b184e046de88cdfcd4f391657e/users/3c24c091db5f474083170fe78dd35b79/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:21:02.704198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec1a63-b093-4c5d-8beb-a010ada2fb27 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:02.704773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec1a63-b093-4c5d-8beb-a010ada2fb27 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:02.704773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec1a63-b093-4c5d-8beb-a010ada2fb27 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:02.706045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 584/1154] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:02.711818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c70598-a3a5-4fc6-96f2-0fa1b06d814b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:02.712248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c70598-a3a5-4fc6-96f2-0fa1b06d814b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:02.712355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c70598-a3a5-4fc6-96f2-0fa1b06d814b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:02.729891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a9c70598-a3a5-4fc6-96f2-0fa1b06d814b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:02.731082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a9c70598-a3a5-4fc6-96f2-0fa1b06d814b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:02.776049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:02.796749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 571/1155] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:03.063049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec1a63-b093-4c5d-8beb-a010ada2fb27 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:03.162322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec1a63-b093-4c5d-8beb-a010ada2fb27 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:03.193548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:03.194663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:03.194936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:03.195234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:03.205910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:21:03.211152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:21:03.212214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:21:03.213152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 585/1156] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:21:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:21:03.218182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c70598-a3a5-4fc6-96f2-0fa1b06d814b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:03.313788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c70598-a3a5-4fc6-96f2-0fa1b06d814b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:03.347420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:03.348011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:03.348287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:03.348558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:03.349209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:03.349507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:03.351130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:03.358940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 572/1157] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:21:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:03.365059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:03.413060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:03.462928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461dc74-ef7d-49cf-a0dd-cdf8a1702b34 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:03.495635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:03.496238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:03.496511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:03.496779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] PATH_INFO: `/v3/users/3c24c091db5f474083170fe78dd35b79` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:03.497873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] RBAC: Authorizing `identity:get_user(user_id=3c24c091db5f474083170fe78dd35b79)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:03.499307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:03.500868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 586/1158] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:21:03 2022] GET /identity/v3/users/3c24c091db5f474083170fe78dd35b79 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:03.506689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:03.550069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:03.605470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2952266f-e46e-4975-a90b-a9f347c4d9ac None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:03.637496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:03.638082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:03.638355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:03.638623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:03.702745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:03.704205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:03.743669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:03.798658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 573/1159] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:21:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:21:03.805135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:03.899657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c9b41d-5d07-4310-88e5-597b19119e5b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:03.915212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:03.933245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e7070f-ae55-441b-bdb8-57a85139ac9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:03.933853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7070f-ae55-441b-bdb8-57a85139ac9b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:03.934123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7070f-ae55-441b-bdb8-57a85139ac9b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:03.934396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e7070f-ae55-441b-bdb8-57a85139ac9b None admin] PATH_INFO: `/v3/projects/7656a1b184e046de88cdfcd4f391657e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:03.935505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e7070f-ae55-441b-bdb8-57a85139ac9b None admin] RBAC: Authorizing `identity:get_project(project_id=7656a1b184e046de88cdfcd4f391657e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:03.936908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e7070f-ae55-441b-bdb8-57a85139ac9b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:03.938505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 587/1160] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:21:03 2022] GET /identity/v3/projects/7656a1b184e046de88cdfcd4f391657e => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:03.945237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:04.039801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6139070-9a6c-4688-8bd3-60132429d376 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:04.072274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44906f38-b0a2-4599-805e-b7f7766387a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:04.072865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44906f38-b0a2-4599-805e-b7f7766387a1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:04.073136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44906f38-b0a2-4599-805e-b7f7766387a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:04.073405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44906f38-b0a2-4599-805e-b7f7766387a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:04.074106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44906f38-b0a2-4599-805e-b7f7766387a1 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=7656a1b184e046de88cdfcd4f391657e, user.id=3c24c091db5f474083170fe78dd35b79) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:04.074406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44906f38-b0a2-4599-805e-b7f7766387a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:04.075871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44906f38-b0a2-4599-805e-b7f7766387a1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:04.081406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 574/1161] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:21:03 2022] GET /identity/v3/role_assignments?user.id=3c24c091db5f474083170fe78dd35b79&scope.project.id=7656a1b184e046de88cdfcd4f391657e&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:06.231548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772cfa3e-1aa2-42a9-9938-114f57eb4887 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:06.232039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772cfa3e-1aa2-42a9-9938-114f57eb4887 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:06.232141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772cfa3e-1aa2-42a9-9938-114f57eb4887 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:06.233397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 588/1162] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:06.238872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10fb03c-fd23-450e-a37c-3412f9965791 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:06.239327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10fb03c-fd23-450e-a37c-3412f9965791 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:06.239453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10fb03c-fd23-450e-a37c-3412f9965791 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:06.252785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:06.258523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f10fb03c-fd23-450e-a37c-3412f9965791 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:06.259741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f10fb03c-fd23-450e-a37c-3412f9965791 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:06.324688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 575/1163] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:06.593262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772cfa3e-1aa2-42a9-9938-114f57eb4887 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:06.691613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772cfa3e-1aa2-42a9-9938-114f57eb4887 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:06.723902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:06.724488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:06.724764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:06.725033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:06.775391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:06.791329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:06.792750 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:06.888044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 589/1164] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:21:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:21:06.893870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10fb03c-fd23-450e-a37c-3412f9965791 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:06.998347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10fb03c-fd23-450e-a37c-3412f9965791 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:07.030561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9be0c6-c6fd-4783-915f-6f5e8dbdbc0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:07.031201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9be0c6-c6fd-4783-915f-6f5e8dbdbc0e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:07.031475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9be0c6-c6fd-4783-915f-6f5e8dbdbc0e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:07.031741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9be0c6-c6fd-4783-915f-6f5e8dbdbc0e None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:07.034275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:07.039886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9be0c6-c6fd-4783-915f-6f5e8dbdbc0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:07.041497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9be0c6-c6fd-4783-915f-6f5e8dbdbc0e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-3a9be0c6-c6fd-4783-915f-6f5e8dbdbc0e None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:21:07.052707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:21:07.054617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:21:07.055567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 576/1165] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:21:06 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:21:07.060480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:07.141014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:07.156348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b67673-7ce8-46c7-8909-3fae367ada03 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:07.189944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee57b5e-fbb5-4909-b62a-8b9161427d78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:07.190524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee57b5e-fbb5-4909-b62a-8b9161427d78 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:07.190793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee57b5e-fbb5-4909-b62a-8b9161427d78 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:07.191088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee57b5e-fbb5-4909-b62a-8b9161427d78 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:07.191714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee57b5e-fbb5-4909-b62a-8b9161427d78 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:07.192008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee57b5e-fbb5-4909-b62a-8b9161427d78 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:07.193348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee57b5e-fbb5-4909-b62a-8b9161427d78 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:07.200224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 590/1166] 10.222.0.33 () {56 vars in 1115 bytes} [Thu Oct 6 04:21:07 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:09.317294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7f050-b888-41e1-99d4-e666932b1d57 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:09.317845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7f050-b888-41e1-99d4-e666932b1d57 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:09.317845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7f050-b888-41e1-99d4-e666932b1d57 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:09.319431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 577/1167] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:09.324741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599bfbf-7f08-4d91-8a3d-fed3d589e5fb None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:09.325166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599bfbf-7f08-4d91-8a3d-fed3d589e5fb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:09.325281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599bfbf-7f08-4d91-8a3d-fed3d589e5fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:09.344687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2599bfbf-7f08-4d91-8a3d-fed3d589e5fb None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:09.345873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2599bfbf-7f08-4d91-8a3d-fed3d589e5fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:09.409911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 591/1168] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:09.679194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7f050-b888-41e1-99d4-e666932b1d57 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:09.776852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7f050-b888-41e1-99d4-e666932b1d57 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:09.808827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cbcde0-8d48-49d4-9f9d-6d1fd7bc35d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:09.809406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cbcde0-8d48-49d4-9f9d-6d1fd7bc35d3 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:09.809680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cbcde0-8d48-49d4-9f9d-6d1fd7bc35d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:09.809947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cbcde0-8d48-49d4-9f9d-6d1fd7bc35d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:09.810606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cbcde0-8d48-49d4-9f9d-6d1fd7bc35d3 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:09.811984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cbcde0-8d48-49d4-9f9d-6d1fd7bc35d3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:09.828048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 578/1169] 10.222.0.33 () {60 vars in 1138 bytes} [Thu Oct 6 04:21:09 2022] POST /identity/v3/domains => generated 267 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:21:12.052989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289bd967-b226-4662-844f-de97cf3d84aa None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:12.055801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289bd967-b226-4662-844f-de97cf3d84aa None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:12.055801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289bd967-b226-4662-844f-de97cf3d84aa None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:12.055801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 592/1170] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:12.061133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60136af-dea2-49a7-b9ca-0bc01ffea9d7 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:12.061525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60136af-dea2-49a7-b9ca-0bc01ffea9d7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:12.061677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60136af-dea2-49a7-b9ca-0bc01ffea9d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:12.080221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c60136af-dea2-49a7-b9ca-0bc01ffea9d7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:12.081398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c60136af-dea2-49a7-b9ca-0bc01ffea9d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:12.130824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:12.147470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 579/1171] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:12.458647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289bd967-b226-4662-844f-de97cf3d84aa None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:12.535988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:12.558104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289bd967-b226-4662-844f-de97cf3d84aa None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:12.590586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ecd29d-bc2d-4a98-ae30-5e800321961a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:12.591217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ecd29d-bc2d-4a98-ae30-5e800321961a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:12.591493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ecd29d-bc2d-4a98-ae30-5e800321961a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:12.591764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ecd29d-bc2d-4a98-ae30-5e800321961a None admin] PATH_INFO: `/v3/domains/bf0d31951a9d4445a581a87ba7514f40` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:12.601706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ecd29d-bc2d-4a98-ae30-5e800321961a None admin] RBAC: Authorizing `identity:get_domain(domain_id=bf0d31951a9d4445a581a87ba7514f40)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:12.603151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ecd29d-bc2d-4a98-ae30-5e800321961a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:12.604692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 593/1172] 10.222.0.33 () {56 vars in 1183 bytes} [Thu Oct 6 04:21:12 2022] GET /identity/v3/domains/bf0d31951a9d4445a581a87ba7514f40 => generated 267 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:12.610468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60136af-dea2-49a7-b9ca-0bc01ffea9d7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:12.711354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60136af-dea2-49a7-b9ca-0bc01ffea9d7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:12.745638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3be59bb-f946-4cfa-9952-2c0095a3a4b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:12.746229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be59bb-f946-4cfa-9952-2c0095a3a4b3 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:12.746503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be59bb-f946-4cfa-9952-2c0095a3a4b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:12.746770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be59bb-f946-4cfa-9952-2c0095a3a4b3 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:12.747448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3be59bb-f946-4cfa-9952-2c0095a3a4b3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:12.748868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3be59bb-f946-4cfa-9952-2c0095a3a4b3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:12.775445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 580/1173] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:21:12 2022] POST /identity/v3/projects => generated 361 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:21:14.970186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32635451-4ee2-46ce-8e54-36efb7c3475c None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:14.970756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32635451-4ee2-46ce-8e54-36efb7c3475c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:14.970756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32635451-4ee2-46ce-8e54-36efb7c3475c None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:14.972107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 594/1174] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:14.977824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f3a7ba-8559-435e-9a25-79ee7fe54a11 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:14.978260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f3a7ba-8559-435e-9a25-79ee7fe54a11 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:14.978355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f3a7ba-8559-435e-9a25-79ee7fe54a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:14.996348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-81f3a7ba-8559-435e-9a25-79ee7fe54a11 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:14.997510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-81f3a7ba-8559-435e-9a25-79ee7fe54a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:15.026495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:15.065741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 581/1175] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:15.336587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32635451-4ee2-46ce-8e54-36efb7c3475c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:15.447381 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32635451-4ee2-46ce-8e54-36efb7c3475c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:15.479350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c293f2-b3c4-4676-9a85-06f855225fc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:15.479952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c293f2-b3c4-4676-9a85-06f855225fc9 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:15.480222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c293f2-b3c4-4676-9a85-06f855225fc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:15.480485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c293f2-b3c4-4676-9a85-06f855225fc9 None admin] PATH_INFO: `/v3/domains/bf0d31951a9d4445a581a87ba7514f40` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:15.481568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c293f2-b3c4-4676-9a85-06f855225fc9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=bf0d31951a9d4445a581a87ba7514f40)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:15.482987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c293f2-b3c4-4676-9a85-06f855225fc9 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:15.484526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 595/1176] 10.222.0.33 () {56 vars in 1183 bytes} [Thu Oct 6 04:21:15 2022] GET /identity/v3/domains/bf0d31951a9d4445a581a87ba7514f40 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:21:15.491172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f3a7ba-8559-435e-9a25-79ee7fe54a11 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:15.586432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f3a7ba-8559-435e-9a25-79ee7fe54a11 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:15.619608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1719914-3390-4c7b-b893-12419814edb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:15.620136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1719914-3390-4c7b-b893-12419814edb7 None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:15.620413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1719914-3390-4c7b-b893-12419814edb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:15.620685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1719914-3390-4c7b-b893-12419814edb7 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:15.621320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1719914-3390-4c7b-b893-12419814edb7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:15.622739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1719914-3390-4c7b-b893-12419814edb7 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:15.643017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 582/1177] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:21:15 2022] POST /identity/v3/users => generated 311 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:21:17.809128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc6266a-1158-4e2c-9b71-c08b560d15c0 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:17.809682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc6266a-1158-4e2c-9b71-c08b560d15c0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:17.809682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc6266a-1158-4e2c-9b71-c08b560d15c0 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:17.811025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 596/1178] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:17.816227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994318f-5f58-4cd3-b41a-a2970d8657ca None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:17.816605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994318f-5f58-4cd3-b41a-a2970d8657ca None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:17.816770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b994318f-5f58-4cd3-b41a-a2970d8657ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:17.835333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b994318f-5f58-4cd3-b41a-a2970d8657ca None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:17.836506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b994318f-5f58-4cd3-b41a-a2970d8657ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:17.853233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:17.902975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 583/1179] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:18.166668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc6266a-1158-4e2c-9b71-c08b560d15c0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:18.175948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:18.266226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc6266a-1158-4e2c-9b71-c08b560d15c0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:18.298320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:18.298930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:18.299235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:18.299503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:18.311635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:21:18.316349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:21:18.319085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:21:18.320664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 597/1180] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:21:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:21:18.324090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b994318f-5f58-4cd3-b41a-a2970d8657ca None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:18.422722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b994318f-5f58-4cd3-b41a-a2970d8657ca None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:18.456374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:18.456966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:18.457243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:18.457515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:18.458167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:18.458468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:18.460894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:18.468681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 584/1181] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:21:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:18.474834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:18.538085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:18.572019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8553e044-9484-4e0c-bd4c-a0100024f05b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:18.603452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:18.604056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:18.604332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:18.604602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] PATH_INFO: `/v3/users/e6f68d00e9b14b8fb2c52731bd5e6a1e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:18.617778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f68d00e9b14b8fb2c52731bd5e6a1e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:18.619216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:18.620771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 598/1182] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:21:18 2022] GET /identity/v3/users/e6f68d00e9b14b8fb2c52731bd5e6a1e => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:18.626473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:18.645746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:18.727376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee8f6bc-09ad-4d80-a265-a75d9ace7c1d None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:18.760104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:18.760697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:18.760971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:18.761239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:18.825348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:18.826773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:18.842495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:18.920711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 585/1183] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:21:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:21:18.926718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:19.023034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977af3eb-a8b5-4d97-bbbe-160031a0e1a2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:19.045275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:19.056691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f923c3d2-b7ea-4273-b5a8-563945874528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:19.057298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f923c3d2-b7ea-4273-b5a8-563945874528 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:19.057569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f923c3d2-b7ea-4273-b5a8-563945874528 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:19.057837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f923c3d2-b7ea-4273-b5a8-563945874528 None admin] PATH_INFO: `/v3/projects/78b1ead204b049ae9f0314f4844c8292` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:19.058915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f923c3d2-b7ea-4273-b5a8-563945874528 None admin] RBAC: Authorizing `identity:get_project(project_id=78b1ead204b049ae9f0314f4844c8292)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:19.060344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f923c3d2-b7ea-4273-b5a8-563945874528 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:19.061931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 599/1184] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:21:18 2022] GET /identity/v3/projects/78b1ead204b049ae9f0314f4844c8292 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:19.068038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:19.165653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feae93ff-33a1-48f7-9001-14bdbe3fc0a4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:19.181627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:19.200263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76ae39f-69ec-40ca-a80b-87d3c6c61968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:19.201119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76ae39f-69ec-40ca-a80b-87d3c6c61968 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:19.201401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76ae39f-69ec-40ca-a80b-87d3c6c61968 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:19.201674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76ae39f-69ec-40ca-a80b-87d3c6c61968 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:19.202393 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76ae39f-69ec-40ca-a80b-87d3c6c61968 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=78b1ead204b049ae9f0314f4844c8292, user.id=e6f68d00e9b14b8fb2c52731bd5e6a1e) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:19.202698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76ae39f-69ec-40ca-a80b-87d3c6c61968 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:19.204155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76ae39f-69ec-40ca-a80b-87d3c6c61968 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:19.210298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 586/1185] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:21:19 2022] GET /identity/v3/role_assignments?user.id=e6f68d00e9b14b8fb2c52731bd5e6a1e&scope.project.id=78b1ead204b049ae9f0314f4844c8292&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:21.302416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5e5a0-217c-4010-9be5-1ccf5ce4fbb3 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:21.303009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5e5a0-217c-4010-9be5-1ccf5ce4fbb3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:21.303009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5e5a0-217c-4010-9be5-1ccf5ce4fbb3 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:21.304361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 600/1186] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:21.309416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d87a0-81a5-4891-a795-5a211ff6ad64 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:21.309842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d87a0-81a5-4891-a795-5a211ff6ad64 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:21.309967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d87a0-81a5-4891-a795-5a211ff6ad64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:21.328660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-157d87a0-81a5-4891-a795-5a211ff6ad64 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:21.329850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-157d87a0-81a5-4891-a795-5a211ff6ad64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:21.396210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 587/1187] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:21.667703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5e5a0-217c-4010-9be5-1ccf5ce4fbb3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:21.765404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5e5a0-217c-4010-9be5-1ccf5ce4fbb3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:21.797023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:21.797649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:21.797922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:21.798191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:21.808636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:21:21.813312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:21:21.814458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:21:21.815544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 601/1188] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:21:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:21:21.821471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157d87a0-81a5-4891-a795-5a211ff6ad64 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:21.922433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157d87a0-81a5-4891-a795-5a211ff6ad64 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:21.933471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:21.956226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:21.956824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:21.957105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:21.957372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:21.958018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:21.958313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:21.959764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:21.967594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 588/1189] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:21:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:21.973521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:22.025563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:22.071042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667f492f-63b8-402f-9604-57c1f1c66ecf None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:22.103517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:22.104123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:22.104394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:22.104661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] PATH_INFO: `/v3/users/e6f68d00e9b14b8fb2c52731bd5e6a1e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:22.107330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] RBAC: Authorizing `identity:get_user(user_id=e6f68d00e9b14b8fb2c52731bd5e6a1e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:22.108759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:22.111973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 602/1190] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:21:21 2022] GET /identity/v3/users/e6f68d00e9b14b8fb2c52731bd5e6a1e => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:22.118324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:22.216745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3e3720-f4f9-41f1-804a-7bd7d61d95a0 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:22.250428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b53a3c-bdcf-4324-9704-42b34e41409a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:22.251067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b53a3c-bdcf-4324-9704-42b34e41409a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:22.251344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b53a3c-bdcf-4324-9704-42b34e41409a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:22.251613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b53a3c-bdcf-4324-9704-42b34e41409a None admin] PATH_INFO: `/v3/projects/78b1ead204b049ae9f0314f4844c8292` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:22.252696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b53a3c-bdcf-4324-9704-42b34e41409a None admin] RBAC: Authorizing `identity:get_project(project_id=78b1ead204b049ae9f0314f4844c8292)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:22.254111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b53a3c-bdcf-4324-9704-42b34e41409a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:22.255743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 589/1191] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:21:22 2022] GET /identity/v3/projects/78b1ead204b049ae9f0314f4844c8292 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:22.261788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:22.357434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718a322-a194-4406-b0a1-e082b65b891c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:22.376706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:22.390952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ff395e-7ae7-44b8-b7c0-08ef49aa5f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:22.391628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ff395e-7ae7-44b8-b7c0-08ef49aa5f06 None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:22.391902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ff395e-7ae7-44b8-b7c0-08ef49aa5f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:22.392173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ff395e-7ae7-44b8-b7c0-08ef49aa5f06 None admin] PATH_INFO: `/v3/projects/78b1ead204b049ae9f0314f4844c8292/users/e6f68d00e9b14b8fb2c52731bd5e6a1e/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:22.394039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ff395e-7ae7-44b8-b7c0-08ef49aa5f06 None admin] RBAC: Authorizing `identity:create_grant(project_id=78b1ead204b049ae9f0314f4844c8292, user_id=e6f68d00e9b14b8fb2c52731bd5e6a1e, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:22.395484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ff395e-7ae7-44b8-b7c0-08ef49aa5f06 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:22.407806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 603/1192] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:21:22 2022] PUT /identity/v3/projects/78b1ead204b049ae9f0314f4844c8292/users/e6f68d00e9b14b8fb2c52731bd5e6a1e/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:21:24.528939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b89fabb-9488-483b-b36b-d6c0a459af45 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:24.529463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b89fabb-9488-483b-b36b-d6c0a459af45 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:24.529463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b89fabb-9488-483b-b36b-d6c0a459af45 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:24.530726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 590/1193] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:21:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:21:24.535980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9ebda-5a70-499e-a899-46d62aaf9e14 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:24.536368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9ebda-5a70-499e-a899-46d62aaf9e14 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:24.536520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9ebda-5a70-499e-a899-46d62aaf9e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:24.554751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-15c9ebda-5a70-499e-a899-46d62aaf9e14 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:21:24.555959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-15c9ebda-5a70-499e-a899-46d62aaf9e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:21:24.619329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 604/1194] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:21:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:21:24.891829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b89fabb-9488-483b-b36b-d6c0a459af45 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:24.934071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:24.992926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b89fabb-9488-483b-b36b-d6c0a459af45 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:25.026822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:25.027464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:25.027739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:25.028008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:25.038302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:21:25.042998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:21:25.044072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:21:25.044955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 591/1195] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:21:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:21:25.050219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9ebda-5a70-499e-a899-46d62aaf9e14 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:25.145435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9ebda-5a70-499e-a899-46d62aaf9e14 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:25.156779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:25.181215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:25.181799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:25.182069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:25.182333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:25.182999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:25.183305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:25.184645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:25.192395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 605/1196] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:21:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:25.198977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:25.294029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:25.297144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940647d0-49fe-4150-92c8-e81bb2a2f70f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:25.331589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:25.332213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:25.332490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:25.332763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] PATH_INFO: `/v3/users/e6f68d00e9b14b8fb2c52731bd5e6a1e` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:25.334052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f68d00e9b14b8fb2c52731bd5e6a1e)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:25.335684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:25.337259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 592/1197] 10.222.0.33 () {56 vars in 1177 bytes} [Thu Oct 6 04:21:25 2022] GET /identity/v3/users/e6f68d00e9b14b8fb2c52731bd5e6a1e => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:25.343062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:25.439753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba4ca6f-5fc6-496e-b61c-832f4cd298fe None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:25.472095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:25.472677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:25.472956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:25.473229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:25.513205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:25.539192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:25.540619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:25.634425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 606/1198] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:21:25 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:21:25.640454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:25.737836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dfec4c-70c1-40a2-b7e5-19e7fe8d4165 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:25.770290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a08e97-3465-4f31-b56d-20e41ec04d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:25.770899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a08e97-3465-4f31-b56d-20e41ec04d9d None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:25.771259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a08e97-3465-4f31-b56d-20e41ec04d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:25.771601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a08e97-3465-4f31-b56d-20e41ec04d9d None admin] PATH_INFO: `/v3/projects/78b1ead204b049ae9f0314f4844c8292` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:25.772705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a08e97-3465-4f31-b56d-20e41ec04d9d None admin] RBAC: Authorizing `identity:get_project(project_id=78b1ead204b049ae9f0314f4844c8292)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:25.774129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a08e97-3465-4f31-b56d-20e41ec04d9d None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:25.775761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 593/1199] 10.222.0.33 () {56 vars in 1186 bytes} [Thu Oct 6 04:21:25 2022] GET /identity/v3/projects/78b1ead204b049ae9f0314f4844c8292 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:25.781725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:21:25.851405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:25.881278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d792ef-4737-4e6a-aa10-e8068fbe1f2f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:21:25.914696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883825b0-fe11-4b08-befd-18c93f5fff0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:21:25.915307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883825b0-fe11-4b08-befd-18c93f5fff0a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:21:25.915584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883825b0-fe11-4b08-befd-18c93f5fff0a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:21:25.915855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883825b0-fe11-4b08-befd-18c93f5fff0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:21:25.916567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883825b0-fe11-4b08-befd-18c93f5fff0a None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=78b1ead204b049ae9f0314f4844c8292, user.id=e6f68d00e9b14b8fb2c52731bd5e6a1e) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:21:25.916872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883825b0-fe11-4b08-befd-18c93f5fff0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:21:25.918273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883825b0-fe11-4b08-befd-18c93f5fff0a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:21:25.923853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 607/1200] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:21:25 2022] GET /identity/v3/role_assignments?user.id=e6f68d00e9b14b8fb2c52731bd5e6a1e&scope.project.id=78b1ead204b049ae9f0314f4844c8292&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:21:35.766223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=82 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:21:35.921313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=32 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:21.587665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5538cb-20d9-4dd0-b3f8-0f997becbfe3 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:21.588312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5538cb-20d9-4dd0-b3f8-0f997becbfe3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:21.588312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5538cb-20d9-4dd0-b3f8-0f997becbfe3 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:21.589591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 594/1201] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:21.595475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3459dd-eb63-4fb1-b399-3e10c90e6d14 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:21.595984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3459dd-eb63-4fb1-b399-3e10c90e6d14 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:21.595984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3459dd-eb63-4fb1-b399-3e10c90e6d14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:21.617498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bb3459dd-eb63-4fb1-b399-3e10c90e6d14 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:21.618711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bb3459dd-eb63-4fb1-b399-3e10c90e6d14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:21.706645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 608/1202] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:22:21.990342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5538cb-20d9-4dd0-b3f8-0f997becbfe3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:22.134760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5538cb-20d9-4dd0-b3f8-0f997becbfe3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:22.178867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:22.179472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:22.179883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:22.180096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:22.191572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:22.193248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:22.209749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:22.210858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:22.212276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:22.212276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:22.212276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:22.212276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 595/1203] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:22.220553 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3459dd-eb63-4fb1-b399-3e10c90e6d14 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:22.346557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3459dd-eb63-4fb1-b399-3e10c90e6d14 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:22.381248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:22.387914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060e92fb-face-4c1a-842b-bd4aa916f9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:22.388533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060e92fb-face-4c1a-842b-bd4aa916f9ea None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:22.388838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060e92fb-face-4c1a-842b-bd4aa916f9ea None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:22.389080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060e92fb-face-4c1a-842b-bd4aa916f9ea None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:22.389807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060e92fb-face-4c1a-842b-bd4aa916f9ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:22.390110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060e92fb-face-4c1a-842b-bd4aa916f9ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:22.391510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060e92fb-face-4c1a-842b-bd4aa916f9ea None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:22.404084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 609/1204] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:22:22.409864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:22.490716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:22.558098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53ff182-be07-4a07-b972-ae558d0aae03 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:22.603050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d88c14b-f181-4d41-8028-082be40d173f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:22.603050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88c14b-f181-4d41-8028-082be40d173f None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:22.603789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88c14b-f181-4d41-8028-082be40d173f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:22.603789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88c14b-f181-4d41-8028-082be40d173f None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:22.604177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d88c14b-f181-4d41-8028-082be40d173f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:22.605620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d88c14b-f181-4d41-8028-082be40d173f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:22.637907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 596/1205] 10.222.0.33 () {60 vars in 1132 bytes} [Thu Oct 6 04:22:22 2022] POST /identity/v3/users => generated 251 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:22:24.939030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bde016-2a36-4c3a-8aee-72b60cdc3da4 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:24.940486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bde016-2a36-4c3a-8aee-72b60cdc3da4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:24.940486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bde016-2a36-4c3a-8aee-72b60cdc3da4 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:24.940926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 610/1206] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:24.947054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809b4679-675d-4677-854b-a5c743b94895 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:24.947512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809b4679-675d-4677-854b-a5c743b94895 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:24.947683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809b4679-675d-4677-854b-a5c743b94895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:24.966698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-809b4679-675d-4677-854b-a5c743b94895 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:24.968034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-809b4679-675d-4677-854b-a5c743b94895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:25.040946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 597/1207] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:22:25.315273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bde016-2a36-4c3a-8aee-72b60cdc3da4 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:25.436564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bde016-2a36-4c3a-8aee-72b60cdc3da4 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:25.476952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:25.477495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:25.477768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:25.478096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:25.491214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:22:25.496589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:25.497668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:25.500136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 611/1208] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:22:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:25.503939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809b4679-675d-4677-854b-a5c743b94895 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:25.547623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:25.620235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809b4679-675d-4677-854b-a5c743b94895 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:25.669705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:25.670384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:25.670787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:25.670945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:25.672028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:25.672476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:25.674004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:25.685147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 598/1209] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:22:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:25.691375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:25.758468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:25.819227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a04017f-8a61-4bfe-9fb0-0b36d18d3cdc None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:25.859411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:25.860049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:25.860317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:25.860632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:25.873850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:25.875584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:25.893438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:25.894763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:25.896673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:25.896673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:25.896673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:25.896673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 612/1210] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:25.900884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:26.027538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:26.030105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a293e-6cbd-4cac-bc6e-0a57a5e1b80e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:26.067743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:26.068478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:26.068759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:26.069033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:26.069672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:26.069982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:26.071389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:26.084228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 599/1211] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:26.089932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:26.196063 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e916b288-a32b-4bf1-9b31-8c786fb6f978 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:26.197738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:26.237034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:26.237899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:26.238212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:26.238594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:26.248438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:26.249983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:22:26.261865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:26.262911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:26.263973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 613/1212] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:26 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:26.269163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:26.387372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab64120e-d9a8-46dc-b9cf-2d35e523b489 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:26.427303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:26.428017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:26.428273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:26.428540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:26.429351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:26.429702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:26.431234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:26.452216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 600/1213] 10.222.0.33 () {56 vars in 1143 bytes} [Thu Oct 6 04:22:26 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:26.459112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:26.570975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1444c7a6-3a92-474d-bd17-bcfb9078ef04 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:26.607237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:26.607838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:26.608573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:26.608573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:26.641306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:26.689428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:26.691115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:26.800288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 614/1214] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:22:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:22:26.806820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:26.827668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:26.934895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f44dd363-1051-4772-b9bb-bd4963236802 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:26.971587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:26.972450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:26.972911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:26.973398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:26.984199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:26.986120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:27.001881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:27.002912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:27.004090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:27.004090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:27.004090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:27.005513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 601/1215] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:27.011356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:27.074857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:27.115459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e34d8c-8959-4ac7-adcd-542f69b6adfd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:27.150608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d450d5-394f-4094-a377-780204955291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:27.151245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d450d5-394f-4094-a377-780204955291 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:27.151520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d450d5-394f-4094-a377-780204955291 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:27.151793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d450d5-394f-4094-a377-780204955291 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:27.152439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d450d5-394f-4094-a377-780204955291 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:27.152741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d450d5-394f-4094-a377-780204955291 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:27.154074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d450d5-394f-4094-a377-780204955291 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:27.165999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 615/1216] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:22:27.172466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:27.260429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:27.295905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b4427d-e509-4f6d-b5c8-f17f5bf0496f None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:27.333769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:27.334449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:27.334731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:27.335041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:27.342065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:27.343549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:22:27.356587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:27.357738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:27.358877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 602/1217] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:22:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:22:27.365867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d450d5-394f-4094-a377-780204955291 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:27.479802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d450d5-394f-4094-a377-780204955291 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:27.507662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:27.520868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eae7a96-768a-4e11-8300-b78b99c2c1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:27.521462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eae7a96-768a-4e11-8300-b78b99c2c1a0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:27.521739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eae7a96-768a-4e11-8300-b78b99c2c1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:27.522020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eae7a96-768a-4e11-8300-b78b99c2c1a0 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:27.522695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eae7a96-768a-4e11-8300-b78b99c2c1a0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:27.523026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eae7a96-768a-4e11-8300-b78b99c2c1a0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:27.525047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eae7a96-768a-4e11-8300-b78b99c2c1a0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:27.538061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 616/1218] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:22:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:27.544342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:27.665300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d146ead5-20fa-44e1-a4a7-f34196a08a83 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:27.698178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:27.705017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998306ec-ca30-494f-8948-a7d68131c3e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:27.705641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998306ec-ca30-494f-8948-a7d68131c3e9 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:27.705906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998306ec-ca30-494f-8948-a7d68131c3e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:27.706179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998306ec-ca30-494f-8948-a7d68131c3e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:27.706900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998306ec-ca30-494f-8948-a7d68131c3e9 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:27.707936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998306ec-ca30-494f-8948-a7d68131c3e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:27.709519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998306ec-ca30-494f-8948-a7d68131c3e9 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:27.715184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 603/1219] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:22:27 2022] GET /identity/v3/role_assignments?user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:30.133134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100e097-1636-4102-afa5-5652010acf0f None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:30.133701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100e097-1636-4102-afa5-5652010acf0f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:30.133701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100e097-1636-4102-afa5-5652010acf0f None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:30.135046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 617/1220] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:30.141385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f6671-f062-4edd-9867-9c911d8a4ce6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:30.141856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f6671-f062-4edd-9867-9c911d8a4ce6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:30.141943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f6671-f062-4edd-9867-9c911d8a4ce6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:30.162739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-2a1f6671-f062-4edd-9867-9c911d8a4ce6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:30.164030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2a1f6671-f062-4edd-9867-9c911d8a4ce6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:30.237726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 604/1221] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:22:30.508703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f100e097-1636-4102-afa5-5652010acf0f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:30.623859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f100e097-1636-4102-afa5-5652010acf0f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:30.663996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:30.664609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:30.664890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:30.665159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:30.675573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:22:30.680329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:30.681502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:30.682579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 618/1222] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:22:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:30.689870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f6671-f062-4edd-9867-9c911d8a4ce6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:30.805436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f6671-f062-4edd-9867-9c911d8a4ce6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:30.832735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:30.845930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:30.846589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:30.847099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:30.847456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:30.849118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] RBAC: Adding query filter params (name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:30.849494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:30.851018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:30.861480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 605/1223] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:22:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:30.867599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:30.919303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:30.986103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2bba8f-0715-4406-b646-55d843a37c76 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:31.031115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:31.031779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:31.032132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:31.032395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:31.042309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:31.043769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:31.062699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:31.063883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:31.065032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:31.065032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:31.065032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:31.065032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 619/1224] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:31.070476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:31.183498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4af1b5-72fc-4dbd-bc1a-c7bccc788def None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:31.220264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:31.220874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:31.221156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:31.221465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:31.222096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:31.222380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:31.224386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:31.237219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 606/1225] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:31.242902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:31.341720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:31.349348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9354e8-5d07-4c55-918f-dd304cb1685b None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:31.384770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:31.385393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:31.385668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:31.385945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:31.393628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:31.395152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:22:31.406353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:31.407548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:31.408752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 620/1226] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:31 2022] GET /identity/v3/users/placement => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 06 04:22:31.413704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:31.462650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:31.520853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7423377f-fac8-42f5-9eca-074a22817fb4 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:31.556214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:31.556893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:31.557170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:31.557466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:31.558186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:31.558494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:31.559969 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:31.577651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 607/1227] 10.222.0.33 () {56 vars in 1143 bytes} [Thu Oct 6 04:22:31 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:31.583760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:31.695943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4da1590-6f97-45e0-bac5-8469d1626593 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:31.733185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:31.733807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:31.734087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:31.734361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:31.746243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:31.747748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:31.765614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:31.766760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:31.768103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:31.768103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:31.768103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:31.768103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 621/1228] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:31.773342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:31.881102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda0cf27-4140-4e01-9557-8490942ed1c1 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:31.883883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:31.920067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:31.920702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:31.920987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:31.921255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:31.921869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:31.922165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:31.923568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:31.937226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 608/1229] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:31.943167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:31.952861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:32.060784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecbc5d8-2ce3-4633-9ca1-0fef2dc238b0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:32.097397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:32.098586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:32.098907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:32.099323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:32.108585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:32.110427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:22:32.124024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:32.124992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:32.125936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 622/1230] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:22:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:32.132395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:32.250148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478746be-8b9c-43ec-a1fc-7426cd20254c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:32.295570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e6c898-a393-4efa-b5f9-bfe8d57494d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:32.296311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e6c898-a393-4efa-b5f9-bfe8d57494d6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:32.296654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e6c898-a393-4efa-b5f9-bfe8d57494d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:32.297013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e6c898-a393-4efa-b5f9-bfe8d57494d6 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:32.297704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e6c898-a393-4efa-b5f9-bfe8d57494d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:32.298006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e6c898-a393-4efa-b5f9-bfe8d57494d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:32.299481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e6c898-a393-4efa-b5f9-bfe8d57494d6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:32.321591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 609/1231] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:22:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:32.323774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:32.404451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:32.443027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6bfe24-3b03-4e3b-907b-20da3c401eb5 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:32.479642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b2304d-f58e-414e-96b5-efa9aa47ce7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:32.480233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2304d-f58e-414e-96b5-efa9aa47ce7d None admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:32.480508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2304d-f58e-414e-96b5-efa9aa47ce7d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:32.480776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b2304d-f58e-414e-96b5-efa9aa47ce7d None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/7e2fe69b3b3b4ba5b55a4698b3ed70b2/roles/bcc8e52e7c184e5984637371b5614639` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:32.496287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b2304d-f58e-414e-96b5-efa9aa47ce7d None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=7e2fe69b3b3b4ba5b55a4698b3ed70b2, role_id=bcc8e52e7c184e5984637371b5614639)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:32.497711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b2304d-f58e-414e-96b5-efa9aa47ce7d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:32.510384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 623/1232] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:22:32 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/7e2fe69b3b3b4ba5b55a4698b3ed70b2/roles/bcc8e52e7c184e5984637371b5614639 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:22:34.841258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679cd4f5-9377-43e0-9d48-050d25602d34 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:34.841939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679cd4f5-9377-43e0-9d48-050d25602d34 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:34.842056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679cd4f5-9377-43e0-9d48-050d25602d34 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:34.843743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 610/1233] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:34.850134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d715fa80-4348-4e3a-8d38-13b8a8f1b680 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:34.850592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d715fa80-4348-4e3a-8d38-13b8a8f1b680 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:34.850694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d715fa80-4348-4e3a-8d38-13b8a8f1b680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:34.869863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d715fa80-4348-4e3a-8d38-13b8a8f1b680 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:34.871103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d715fa80-4348-4e3a-8d38-13b8a8f1b680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:34.952252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 624/1234] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:22:35.281962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679cd4f5-9377-43e0-9d48-050d25602d34 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:35.312721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:35.402085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679cd4f5-9377-43e0-9d48-050d25602d34 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:35.436941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:35.437568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:35.437845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:35.438114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:35.452791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:22:35.458662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:35.459865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:35.460921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 611/1235] 10.222.0.33 () {56 vars in 1102 bytes} [Thu Oct 6 04:22:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:35.467255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d715fa80-4348-4e3a-8d38-13b8a8f1b680 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:35.523432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:35.581903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d715fa80-4348-4e3a-8d38-13b8a8f1b680 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:35.618871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:35.619582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:35.619866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:35.620164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:35.621086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] RBAC: Adding query filter params (name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:35.621393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:35.622903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:35.635579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 625/1236] 10.222.0.33 () {56 vars in 1103 bytes} [Thu Oct 6 04:22:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:35.641721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:35.744488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:35.766477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13f564f-2ebb-469b-a941-f84cdd70838a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:35.808175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:35.808846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:35.809091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:35.809371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:35.821161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:35.822617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:35.840324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:35.841901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:35.842837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:35.842837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:35.842837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:35.842837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 612/1237] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:35.848204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:35.962364 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4336d4ab-c591-4cb0-9831-42e89d28ac79 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:35.974183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:36.014176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:36.014932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:36.015352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:36.015740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:36.016914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:36.017415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:36.019506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:36.032558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 626/1238] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:36.038870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:36.150904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb97ac2-766d-4049-9af4-faa91358f1fb None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:36.185228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:36.185887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:36.186279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:36.186606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] PATH_INFO: `/v3/users/placement` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:36.189911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:36.196033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:36.197833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:22:36.210712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:36.211797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:36.212869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 613/1239] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:36 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:36.217760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:36.329355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc88ba86-9cff-4be5-975d-769dc857adaf None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:36.376237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:36.376920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:36.377221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:36.377545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:36.378338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:36.378687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:36.380585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:36.401137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 627/1240] 10.222.0.33 () {56 vars in 1143 bytes} [Thu Oct 6 04:22:36 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:36.407176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:36.524477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0254902-8b31-4f41-8db0-b78397840aae None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:36.559522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:36.560130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:36.560407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:36.560722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:36.628705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:36.636259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:36.637901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:36.748167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 614/1241] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:22:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:22:36.754852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:36.782747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:36.865953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db8467-a7f5-4580-8df6-5d5a747db28c None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:36.907393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:36.908179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:36.908555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:36.908994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:36.925062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:36.927912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:36.949387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:36.956188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:36.957875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:36.957875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:36.957875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:36.957875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 628/1242] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:36.958119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:37.062536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:37.069621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b35c47f-17d7-46aa-95a1-3dd049075e1e None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:37.107432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:37.108043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:37.108320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:37.108589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:37.109220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:37.109519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:37.111252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:37.124858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 615/1243] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:37.133083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:37.220439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:37.250829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c017de-8c01-45c5-abcf-3b8b7d80bd16 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:37.291445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:37.292247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:37.292538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:37.292818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:37.301610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:37.303308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:22:37.316200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:37.317596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:37.318661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 629/1244] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:22:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:37.325066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:37.443817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc43f9a4-0986-4db8-af20-b15ce064f04b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:37.482250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3171b2b-a994-4e1a-aa39-778d491a1697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:37.482763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3171b2b-a994-4e1a-aa39-778d491a1697 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:37.483113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3171b2b-a994-4e1a-aa39-778d491a1697 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:37.483446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3171b2b-a994-4e1a-aa39-778d491a1697 None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:37.484333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3171b2b-a994-4e1a-aa39-778d491a1697 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:37.484600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3171b2b-a994-4e1a-aa39-778d491a1697 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:37.486547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3171b2b-a994-4e1a-aa39-778d491a1697 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:37.500000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 616/1245] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:22:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:37.506885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:37.619319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaae0da-88d8-42fb-94ee-9085cb74e30e None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:37.656177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538edf66-6514-4699-bcb8-53f424a95899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:37.656859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538edf66-6514-4699-bcb8-53f424a95899 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:37.657156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538edf66-6514-4699-bcb8-53f424a95899 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:37.657430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538edf66-6514-4699-bcb8-53f424a95899 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:37.658221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538edf66-6514-4699-bcb8-53f424a95899 None admin] RBAC: Adding query filter params (role.id=bcc8e52e7c184e5984637371b5614639, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:37.658529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538edf66-6514-4699-bcb8-53f424a95899 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:37.660577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538edf66-6514-4699-bcb8-53f424a95899 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:37.667639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 630/1246] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:22:37 2022] GET /identity/v3/role_assignments?user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=bcc8e52e7c184e5984637371b5614639 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:40.026452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68839885-052e-44ec-8b15-0fb73dde0c41 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:40.027056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68839885-052e-44ec-8b15-0fb73dde0c41 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:40.027153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68839885-052e-44ec-8b15-0fb73dde0c41 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:40.028488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 617/1247] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:40.034082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b24ae-725e-48da-9b4c-4dd65ffb1d9a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:40.034544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b24ae-725e-48da-9b4c-4dd65ffb1d9a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:40.034708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7b24ae-725e-48da-9b4c-4dd65ffb1d9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:40.041628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:40.057002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7a7b24ae-725e-48da-9b4c-4dd65ffb1d9a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:40.058535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7a7b24ae-725e-48da-9b4c-4dd65ffb1d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:40.142986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 631/1248] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:22:40.438839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68839885-052e-44ec-8b15-0fb73dde0c41 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:40.459751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:40.554364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68839885-052e-44ec-8b15-0fb73dde0c41 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:40.592096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:40.592717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:40.592991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:40.593257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:40.603668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:22:40.608390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:40.609550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:40.610450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 618/1249] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:22:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:40.616707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b24ae-725e-48da-9b4c-4dd65ffb1d9a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:40.727787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7b24ae-725e-48da-9b4c-4dd65ffb1d9a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:40.761936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:40.769661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:40.770633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:40.771137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:40.771721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:40.772529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:40.772919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:40.775292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:40.784541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 632/1250] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:22:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:40.791180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:40.882322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:40.910379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ec65e3-3b3e-4f0a-b67d-02622fa7049c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:40.952065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:40.952955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:40.953158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:40.953433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:40.965260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:40.966714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:40.984198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:40.985373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:40.986707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:40.986707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:40.986707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:40.986707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 619/1251] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:40.991666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:41.106385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f7ee9f-25c2-4d5e-a874-25230a8ee2f7 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:41.144089 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:41.145298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:41.145573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:41.145869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:41.146556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:41.146895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:41.148376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:41.161283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 633/1252] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:41.167304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:41.281774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4b40d7-fb92-4a99-8e93-6760cb7b4cfe None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:41.310771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:41.320632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:41.321338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:41.321692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:41.321993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:41.330251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:41.331933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:22:41.343742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:41.344754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:41.345733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:41.346349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 620/1253] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:41 2022] GET /identity/v3/users/placement => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:41.353541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:41.415765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:41.465174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effa873-b24d-46cd-aee5-af8c49295a5d None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:41.506669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:41.507307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:41.507581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:41.507851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:41.508577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:41.508878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:41.510294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:41.528690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 634/1254] 10.222.0.33 () {56 vars in 1143 bytes} [Thu Oct 6 04:22:41 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:41.534777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:41.642300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c6ee5b-47de-4889-bf9d-7db427f9f458 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:41.677885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:41.678505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:41.678780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:41.679081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:41.737421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:41.751113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:41.752597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:41.855847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 621/1255] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:22:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:22:41.862088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:41.983257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab4e5e-61ee-418e-b120-38b6d54bbea2 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:41.997258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:42.019668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:42.020289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:42.020573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:42.020835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:42.033571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:42.033571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:42.047809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:42.048976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:42.050072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:42.050072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:42.050072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:42.050072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 635/1256] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:42.055566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:42.149093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:42.161862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99080b5-d05c-4736-9323-81a0261bea2c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:42.195516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:42.196110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:42.196386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:42.196655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:42.197283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:42.197583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:42.198941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:42.209892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 622/1257] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:42.215598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:42.312223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ac98f3-8daa-4625-adfa-9ce704a9f692 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:42.344254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:42.344865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:42.345139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:42.345409 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:42.352428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:42.353841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:22:42.365635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:42.366710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:42.368027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 636/1258] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:22:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:42.372565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:42.469811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4ea1b5-5d79-4522-8308-196481786ee9 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:42.502933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42451543-8e49-4cb1-a89e-25de6ac1cf0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:42.503581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42451543-8e49-4cb1-a89e-25de6ac1cf0c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:42.503881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42451543-8e49-4cb1-a89e-25de6ac1cf0c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:42.504161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42451543-8e49-4cb1-a89e-25de6ac1cf0c None admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:42.504841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42451543-8e49-4cb1-a89e-25de6ac1cf0c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:42.505168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42451543-8e49-4cb1-a89e-25de6ac1cf0c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:42.506586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42451543-8e49-4cb1-a89e-25de6ac1cf0c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:42.518235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 623/1259] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:22:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:42.524847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:42.550151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:42.623006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e39dcee-c774-4957-b362-fca7d1bba261 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:42.655443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d2e80-386f-4140-99ed-e8c56222d228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:42.656014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d2e80-386f-4140-99ed-e8c56222d228 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:42.656289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d2e80-386f-4140-99ed-e8c56222d228 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:42.656559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d2e80-386f-4140-99ed-e8c56222d228 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:42.657260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d2e80-386f-4140-99ed-e8c56222d228 None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:42.657559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d2e80-386f-4140-99ed-e8c56222d228 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:42.658982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d2e80-386f-4140-99ed-e8c56222d228 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:42.664319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 637/1260] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:22:42 2022] GET /identity/v3/role_assignments?user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:44.821748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77571ba4-fc95-4395-9c8e-514b19c22c98 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:44.822307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77571ba4-fc95-4395-9c8e-514b19c22c98 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:44.822307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77571ba4-fc95-4395-9c8e-514b19c22c98 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:44.823660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 624/1261] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:44.828744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f9e2d5-c7c7-4775-8f8c-cd64e8cda458 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:44.829150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f9e2d5-c7c7-4775-8f8c-cd64e8cda458 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:44.829295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f9e2d5-c7c7-4775-8f8c-cd64e8cda458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:44.847780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a3f9e2d5-c7c7-4775-8f8c-cd64e8cda458 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:44.849765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a3f9e2d5-c7c7-4775-8f8c-cd64e8cda458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:44.922273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 638/1262] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:22:45.213880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77571ba4-fc95-4395-9c8e-514b19c22c98 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:45.224851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:45.313213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77571ba4-fc95-4395-9c8e-514b19c22c98 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:45.346609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:45.347260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:45.347537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:45.347806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:45.357748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:22:45.362706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:45.363875 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:45.365249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 625/1263] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:22:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:45.370062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f9e2d5-c7c7-4775-8f8c-cd64e8cda458 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:45.389217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:45.476061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f9e2d5-c7c7-4775-8f8c-cd64e8cda458 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:45.513258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:45.513868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:45.514142 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:45.514410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:45.515098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:45.515565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:45.516920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:45.525558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 639/1264] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:22:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:45.531467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:45.596831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:45.631041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67b9bb1-dcfe-4c75-94d3-1ac67c71dd9c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:45.662494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:45.663126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:45.663403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:45.663672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:45.673723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:45.675180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:45.689290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:45.690391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:45.691528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:45.691528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:45.691528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:45.691528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 626/1265] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:45.699221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:45.783896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:45.803592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55765f05-e1a6-49fc-897c-3fb3b504c139 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:45.835161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:45.835728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:45.836005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:45.836272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:45.836895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:45.837191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:45.838541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:45.850352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 640/1266] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:45.856178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:45.960204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80d891f-b37b-4f45-bf39-09c469cb89f1 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:45.979699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:45.994546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:45.995197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:45.995476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:45.995746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] PATH_INFO: `/v3/users/placement` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:46.002901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:46.004495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:22:46.019703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:46.020943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:46.023407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 627/1267] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:45 2022] GET /identity/v3/users/placement => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:46.027013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:46.135726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04226d1d-ec26-439e-9366-582127e5fb0a None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:46.168067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:46.168649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:46.168924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:46.169192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:46.169900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:46.170202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:46.171647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:46.180569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:46.189464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 641/1268] 10.222.0.33 () {56 vars in 1143 bytes} [Thu Oct 6 04:22:46 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:46.195523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:46.295250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d677b72-e2a6-4358-aad4-746f3a4ee68a None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:46.327333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:46.327939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:46.328212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:46.328478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:46.340017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:46.341489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:46.357266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:46.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:46.359592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:46.359592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:46.359592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:46.359592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 628/1269] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:46.364389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:46.459544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f17a6d3-c4f9-4879-a2aa-020947ec820f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:46.493246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:46.493818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:46.494089 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:46.494357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:46.494993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:46.495293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:46.496636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:46.507430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 642/1270] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:46.513025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:46.544824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:46.611390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e9b604-5a4a-43f1-9dd1-b4bf88d8aefc None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:46.644189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:46.645395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:46.645734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:46.646010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:46.652397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:46.653816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:22:46.664800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:46.665852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:46.666997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 629/1271] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:22:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:46.671584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:46.727770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:46.768050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f23f3c-835c-4f58-b176-5d588c744ee0 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:46.800560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2b130a-dc2c-4e8f-b754-81a286e5df62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:46.801168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b130a-dc2c-4e8f-b754-81a286e5df62 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:46.801432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b130a-dc2c-4e8f-b754-81a286e5df62 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:46.801696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b130a-dc2c-4e8f-b754-81a286e5df62 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:46.802366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2b130a-dc2c-4e8f-b754-81a286e5df62 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:46.802662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2b130a-dc2c-4e8f-b754-81a286e5df62 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:46.804086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2b130a-dc2c-4e8f-b754-81a286e5df62 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:46.815405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 643/1272] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:22:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:46.822101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:46.915026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:46.924618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5804fab-1fdd-45e1-a861-14eeded9bb0c None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:46.958849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a006d-9b17-430d-bc9e-061b322925ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:46.959514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a006d-9b17-430d-bc9e-061b322925ef None admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:46.959791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a006d-9b17-430d-bc9e-061b322925ef None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:46.960063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a006d-9b17-430d-bc9e-061b322925ef None admin] PATH_INFO: `/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/7e2fe69b3b3b4ba5b55a4698b3ed70b2/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:46.961962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a006d-9b17-430d-bc9e-061b322925ef None admin] RBAC: Authorizing `identity:create_grant(project_id=dd80852f67f74db3be0a04ef0a28ec38, user_id=7e2fe69b3b3b4ba5b55a4698b3ed70b2, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:46.963422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a006d-9b17-430d-bc9e-061b322925ef None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:46.977161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 630/1273] 10.222.0.33 () {58 vars in 1439 bytes} [Thu Oct 6 04:22:46 2022] PUT /identity/v3/projects/dd80852f67f74db3be0a04ef0a28ec38/users/7e2fe69b3b3b4ba5b55a4698b3ed70b2/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:22:49.103593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcaabc-623b-42a4-bf23-d309ee1d5c8c None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:49.104127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcaabc-623b-42a4-bf23-d309ee1d5c8c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:49.104127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bcaabc-623b-42a4-bf23-d309ee1d5c8c None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:49.105442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 644/1274] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:49.110515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b4b376-e69c-4eaa-8a13-d497baf3c87c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:49.111206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b4b376-e69c-4eaa-8a13-d497baf3c87c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:49.111364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b4b376-e69c-4eaa-8a13-d497baf3c87c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:49.129954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b4b4b376-e69c-4eaa-8a13-d497baf3c87c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:49.131158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b4b4b376-e69c-4eaa-8a13-d497baf3c87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:49.195094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 631/1275] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:22:49.461866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcaabc-623b-42a4-bf23-d309ee1d5c8c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:49.560552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bcaabc-623b-42a4-bf23-d309ee1d5c8c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:49.577008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:49.592933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:49.593541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:49.593814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:49.594077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:49.604210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 06 04:22:49.608864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:49.610009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:49.611234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 645/1276] 10.222.0.33 () {56 vars in 1096 bytes} [Thu Oct 6 04:22:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:49.617363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b4b376-e69c-4eaa-8a13-d497baf3c87c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:49.698116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:49.717467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b4b376-e69c-4eaa-8a13-d497baf3c87c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:49.752264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:49.752855 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:49.753133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:49.753405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:49.754088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:49.754783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:49.756189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:49.764024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 632/1277] 10.222.0.33 () {56 vars in 1099 bytes} [Thu Oct 6 04:22:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:49.769731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:49.865642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0023d6-987f-47cc-b3ca-5c3e8511c885 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:49.899118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:49.899703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:49.899990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:49.900284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:49.911734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:49.913377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:49.931714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:49.933012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:49.934127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:49.934127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:49.934127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:49.934127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 646/1278] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:49.940491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:50.042162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c83083-05cb-43c4-a8d3-af361a617c62 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:50.074142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:50.074736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:50.075040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:50.075316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:50.075942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:50.076241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:50.077611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:50.089546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 633/1279] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:50.095255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:50.125671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:50.193732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a76c2-d49a-4906-b4c0-064d87b7469f None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:50.226893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:50.227543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:50.227816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:50.228084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] PATH_INFO: `/v3/users/placement` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:50.236968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:50.238404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 06 04:22:50.249178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:50.250291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:50.251325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 647/1280] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:50 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:50.256335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:50.260104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:50.355074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab21e6-4812-4260-9a18-de1ccb495ece None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:50.391027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:50.391627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:50.391905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:50.392184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:50.392899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:50.393203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:50.394678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:50.412060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 634/1281] 10.222.0.33 () {56 vars in 1143 bytes} [Thu Oct 6 04:22:50 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:50.417943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:50.506365 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:50.515771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83582c9d-f74b-49ed-9b78-da71a6f499eb None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:50.547926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:50.548510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:50.548780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:50.549048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:50.615640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:50.616988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:50.699747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:50.711021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 648/1282] 10.222.0.33 () {58 vars in 1282 bytes} [Thu Oct 6 04:22:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 06 04:22:50.716734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:50.777644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:50.817212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f2ef9f-6572-43b9-9dbd-8c778f3e183b None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:50.851825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:50.852444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:50.852719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:50.852987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:50.863271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:50.865230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 06 04:22:50.879923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:50.881069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 06 04:22:50.882148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 06 04:22:50.882148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 06 04:22:50.882148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:50.882148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 635/1283] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:22:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:50.886678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:50.990106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94802cb5-6802-49bb-a475-ebc706ec9775 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:51.023612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:51.024201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:51.024480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:51.024749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:51.025376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:51.025675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:51.027046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:51.038247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 649/1284] 10.222.0.33 () {56 vars in 1109 bytes} [Thu Oct 6 04:22:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:51.043863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:51.144171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26cc737-5500-4fcc-bd2a-c1e66a3e56b3 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:51.157756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:51.179694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:51.180309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:51.180588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:51.180860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:51.187338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:51.188777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 06 04:22:51.200351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:51.201418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: ERROR keystone.server.flask.application Oct 06 04:22:51.202674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 636/1285] 10.222.0.33 () {56 vars in 1111 bytes} [Thu Oct 6 04:22:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:51.207194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:51.256757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:51.304898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d76ac7-e05f-4648-a935-16c5fb3c1001 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:51.336191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bb3910-519b-4d91-91e0-2e7e7c099a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:51.336782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bb3910-519b-4d91-91e0-2e7e7c099a18 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:51.337056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bb3910-519b-4d91-91e0-2e7e7c099a18 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:51.337326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bb3910-519b-4d91-91e0-2e7e7c099a18 None admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:51.337997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bb3910-519b-4d91-91e0-2e7e7c099a18 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:51.338294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bb3910-519b-4d91-91e0-2e7e7c099a18 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:51.339726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bb3910-519b-4d91-91e0-2e7e7c099a18 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:51.351652 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 650/1286] 10.222.0.33 () {56 vars in 1148 bytes} [Thu Oct 6 04:22:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:51.357638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:51.455999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40eef4-5163-43bd-ab2a-252446024e55 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:51.488722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb72516e-7290-4154-a2f9-ae0724d524ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:51.489305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb72516e-7290-4154-a2f9-ae0724d524ef None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:51.489579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb72516e-7290-4154-a2f9-ae0724d524ef None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:51.489847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb72516e-7290-4154-a2f9-ae0724d524ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:51.490545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb72516e-7290-4154-a2f9-ae0724d524ef None admin] RBAC: Adding query filter params (role.id=496e1df1a0c74115a98f6bbb36196a95, scope.project.id=dd80852f67f74db3be0a04ef0a28ec38, user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:51.490845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb72516e-7290-4154-a2f9-ae0724d524ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:51.492286 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb72516e-7290-4154-a2f9-ae0724d524ef None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:51.497782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 637/1287] 10.222.0.33 () {56 vars in 1374 bytes} [Thu Oct 6 04:22:51 2022] GET /identity/v3/role_assignments?user.id=7e2fe69b3b3b4ba5b55a4698b3ed70b2&scope.project.id=dd80852f67f74db3be0a04ef0a28ec38&role.id=496e1df1a0c74115a98f6bbb36196a95 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:53.782285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e745b-6375-4f10-ba62-668d460bcc54 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:53.782830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e745b-6375-4f10-ba62-668d460bcc54 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:53.782830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e745b-6375-4f10-ba62-668d460bcc54 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:53.784182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 651/1288] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:53.790284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf3dab-3c45-4549-aabd-bff02f16a4e5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:53.790833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf3dab-3c45-4549-aabd-bff02f16a4e5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:53.790833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf3dab-3c45-4549-aabd-bff02f16a4e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:53.809185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e6bf3dab-3c45-4549-aabd-bff02f16a4e5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:53.810366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e6bf3dab-3c45-4549-aabd-bff02f16a4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:53.840909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:53.879459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 638/1289] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 06 04:22:54.173840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122e745b-6375-4f10-ba62-668d460bcc54 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:54.274108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122e745b-6375-4f10-ba62-668d460bcc54 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:54.285009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:54.308422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:54.309034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:54.309305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:54.309568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:54.310133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:54.311505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:54.320378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:54.321686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:54.322842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 652/1290] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:22:54 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:54.327422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bf3dab-3c45-4549-aabd-bff02f16a4e5 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:54.424754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bf3dab-3c45-4549-aabd-bff02f16a4e5 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:54.458531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af74fa5f-a3b5-4eff-aa2b-115f48fbe56e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:54.459158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af74fa5f-a3b5-4eff-aa2b-115f48fbe56e None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:54.459439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af74fa5f-a3b5-4eff-aa2b-115f48fbe56e None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:54.459710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af74fa5f-a3b5-4eff-aa2b-115f48fbe56e None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:54.460337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af74fa5f-a3b5-4eff-aa2b-115f48fbe56e None admin] RBAC: Adding query filter params (name=placement) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:54.460640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af74fa5f-a3b5-4eff-aa2b-115f48fbe56e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:54.462302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af74fa5f-a3b5-4eff-aa2b-115f48fbe56e None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:54.467027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 639/1291] 10.222.0.33 () {56 vars in 1116 bytes} [Thu Oct 6 04:22:54 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:54.472771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:54.570349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53764c62-a72c-4cae-82d5-90f119657818 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:54.602052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d1707d-f10a-452a-bdbd-fb061ec48538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:54.602632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d1707d-f10a-452a-bdbd-fb061ec48538 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:54.602905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d1707d-f10a-452a-bdbd-fb061ec48538 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:54.603203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d1707d-f10a-452a-bdbd-fb061ec48538 None admin] PATH_INFO: `/v3/services` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:54.603818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d1707d-f10a-452a-bdbd-fb061ec48538 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:22:54.604111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d1707d-f10a-452a-bdbd-fb061ec48538 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:54.605452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d1707d-f10a-452a-bdbd-fb061ec48538 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:54.609712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 653/1292] 10.222.0.33 () {56 vars in 1116 bytes} [Thu Oct 6 04:22:54 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:22:56.709104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab88a8-9097-45b9-8673-dc12d3f82e17 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:56.709640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab88a8-9097-45b9-8673-dc12d3f82e17 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:56.709640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfab88a8-9097-45b9-8673-dc12d3f82e17 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:56.710992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 640/1293] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:56.716203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e83c3-0803-4772-8f3a-dbb5aa1cfc5d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:56.716647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e83c3-0803-4772-8f3a-dbb5aa1cfc5d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:56.716740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e83c3-0803-4772-8f3a-dbb5aa1cfc5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:56.735047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ae3e83c3-0803-4772-8f3a-dbb5aa1cfc5d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:56.736477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ae3e83c3-0803-4772-8f3a-dbb5aa1cfc5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:56.764405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:56.801522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 654/1294] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:22:57.067751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab88a8-9097-45b9-8673-dc12d3f82e17 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:57.098590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:22:57.168416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfab88a8-9097-45b9-8673-dc12d3f82e17 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:57.200935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621cd94d-0a51-4cf3-8155-7d92620b937b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:57.201515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621cd94d-0a51-4cf3-8155-7d92620b937b None admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:57.201786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621cd94d-0a51-4cf3-8155-7d92620b937b None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:57.202051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621cd94d-0a51-4cf3-8155-7d92620b937b None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:57.202690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621cd94d-0a51-4cf3-8155-7d92620b937b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:57.204069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621cd94d-0a51-4cf3-8155-7d92620b937b None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:57.219322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 641/1295] 10.222.0.33 () {60 vars in 1141 bytes} [Thu Oct 6 04:22:57 2022] POST /identity/v3/services => generated 246 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:22:59.346563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a8e66-f983-47a2-807f-778378dcb530 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:59.347172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a8e66-f983-47a2-807f-778378dcb530 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:59.347172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a8e66-f983-47a2-807f-778378dcb530 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:59.349129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 655/1296] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:22:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:22:59.354280 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca84c3a-210c-4447-838d-d0b65287d4ae None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:59.354729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca84c3a-210c-4447-838d-d0b65287d4ae None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:59.354824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca84c3a-210c-4447-838d-d0b65287d4ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:59.373470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-eca84c3a-210c-4447-838d-d0b65287d4ae None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:22:59.374651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-eca84c3a-210c-4447-838d-d0b65287d4ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:22:59.451074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 642/1297] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:22:59 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:22:59.743966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a8e66-f983-47a2-807f-778378dcb530 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:59.845902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a8e66-f983-47a2-807f-778378dcb530 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:22:59.881333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:22:59.881944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:22:59.882220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:22:59.882492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] PATH_INFO: `/v3/services/placement` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:22:59.883092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:22:59.884439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:22:59.893212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:22:59.894830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:22:59.896150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 656/1298] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:22:59 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:22:59.901523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca84c3a-210c-4447-838d-d0b65287d4ae None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:22:59.923348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:23:00.015580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca84c3a-210c-4447-838d-d0b65287d4ae None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:23:00.050143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:23:00.051207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:00.051495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:00.051771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:00.052399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:23:00.052703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:23:00.054077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:23:00.059370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 643/1299] 10.222.0.33 () {56 vars in 1116 bytes} [Thu Oct 6 04:22:59 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:23:00.065535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:23:00.097775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:23:00.165117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1118b4-4ed3-41bd-8675-635b39d21dcd None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:23:00.196904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ebd4be-6b1f-44ac-b2ad-ad49221de9cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:23:00.197486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ebd4be-6b1f-44ac-b2ad-ad49221de9cf None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:00.197757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ebd4be-6b1f-44ac-b2ad-ad49221de9cf None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:00.198023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ebd4be-6b1f-44ac-b2ad-ad49221de9cf None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:00.198701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ebd4be-6b1f-44ac-b2ad-ad49221de9cf None admin] RBAC: Adding query filter params (interface=public, service_id=580920c27a4d422b830911246bf044ae, region_id=RegionOne) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:23:00.199024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ebd4be-6b1f-44ac-b2ad-ad49221de9cf None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:23:00.200363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ebd4be-6b1f-44ac-b2ad-ad49221de9cf None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:23:00.205123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 657/1300] 10.222.0.33 () {56 vars in 1251 bytes} [Thu Oct 6 04:23:00 2022] GET /identity/v3/endpoints?service_id=580920c27a4d422b830911246bf044ae&interface=public®ion_id=RegionOne => generated 194 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:23:00.211487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:23:00.291532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:23:00.309364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f067eea-c8c0-4cbd-9905-ac6f59caf7d6 None admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:23:00.343335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f031ca-4386-4f3c-b64e-4e2ac7099cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:23:00.343930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f031ca-4386-4f3c-b64e-4e2ac7099cd5 None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:00.344206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f031ca-4386-4f3c-b64e-4e2ac7099cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:00.344477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f031ca-4386-4f3c-b64e-4e2ac7099cd5 None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:00.345072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f031ca-4386-4f3c-b64e-4e2ac7099cd5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:23:00.346431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f031ca-4386-4f3c-b64e-4e2ac7099cd5 None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:23:00.353205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 644/1301] 10.222.0.33 () {56 vars in 1087 bytes} [Thu Oct 6 04:23:00 2022] GET /identity/v3/services => generated 2191 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:23:02.475607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d16771-0420-4f92-b536-b18d18554e73 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:02.476135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d16771-0420-4f92-b536-b18d18554e73 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:02.476135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d16771-0420-4f92-b536-b18d18554e73 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:02.477377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 658/1302] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:23:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:23:02.482506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2806550-3632-4a10-b17c-b90cb34201f7 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:02.482893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2806550-3632-4a10-b17c-b90cb34201f7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:02.483081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2806550-3632-4a10-b17c-b90cb34201f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:02.501396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a2806550-3632-4a10-b17c-b90cb34201f7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:23:02.502555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a2806550-3632-4a10-b17c-b90cb34201f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:23:02.567632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 645/1303] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:23:02 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 06 04:23:02.844921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d16771-0420-4f92-b536-b18d18554e73 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:23:02.947682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:23:02.955898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d16771-0420-4f92-b536-b18d18554e73 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:23:02.991890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:23:02.992506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:02.992779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:02.993046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:02.993616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:23:02.994990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 06 04:23:03.003449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 06 04:23:03.005111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.get_or_create( Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application with Lock( Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self._enter() Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application return self.creator() Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application created_value = creator( Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: ERROR keystone.server.flask.application Oct 06 04:23:03.006197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 659/1304] 10.222.0.33 () {56 vars in 1117 bytes} [Thu Oct 6 04:23:02 2022] GET /identity/v3/services/placement => generated 90 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 06 04:23:03.011103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2806550-3632-4a10-b17c-b90cb34201f7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:23:03.085207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:23:03.107894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2806550-3632-4a10-b17c-b90cb34201f7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:23:03.140959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb249e4-2bfd-4435-91c3-80b72656171a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:23:03.141550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb249e4-2bfd-4435-91c3-80b72656171a None admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:03.141827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb249e4-2bfd-4435-91c3-80b72656171a None admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:03.142097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb249e4-2bfd-4435-91c3-80b72656171a None admin] PATH_INFO: `/v3/services` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:03.142716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb249e4-2bfd-4435-91c3-80b72656171a None admin] RBAC: Adding query filter params (name=placement) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:23:03.143039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb249e4-2bfd-4435-91c3-80b72656171a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:23:03.144456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb249e4-2bfd-4435-91c3-80b72656171a None admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:23:03.149421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 646/1305] 10.222.0.33 () {56 vars in 1116 bytes} [Thu Oct 6 04:23:03 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:23:03.155212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:23:03.254475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf627d2-d121-4185-86ba-8b5485473903 None admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:23:03.287004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63e14d4-8aeb-4013-a97a-29f84089a624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:23:03.287587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63e14d4-8aeb-4013-a97a-29f84089a624 None admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:23:03.287858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63e14d4-8aeb-4013-a97a-29f84089a624 None admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:23:03.288123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63e14d4-8aeb-4013-a97a-29f84089a624 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:23:03.288767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63e14d4-8aeb-4013-a97a-29f84089a624 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:23:03.290101 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63e14d4-8aeb-4013-a97a-29f84089a624 None admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:23:03.308614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 660/1306] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:23:03 2022] POST /identity/v3/endpoints => generated 331 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:23:13.146485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=30 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:23:13.304625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:07.474303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c8f69-ce3a-4ffc-acfc-9f9b44a980dc None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:07.474782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c8f69-ce3a-4ffc-acfc-9f9b44a980dc None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:07.474869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c8f69-ce3a-4ffc-acfc-9f9b44a980dc None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:07.476814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 647/1307] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:07.482272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572d5ae-746d-417f-9e06-609969cbdde8 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:07.482658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572d5ae-746d-417f-9e06-609969cbdde8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:07.482826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7572d5ae-746d-417f-9e06-609969cbdde8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:07.649587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7572d5ae-746d-417f-9e06-609969cbdde8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:07.652084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7572d5ae-746d-417f-9e06-609969cbdde8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:07.768357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 661/1308] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:24:08.065814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8664dbcc-1c10-4e36-820b-98e9746d3e11 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:08.066128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8664dbcc-1c10-4e36-820b-98e9746d3e11 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:08.066362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8664dbcc-1c10-4e36-820b-98e9746d3e11 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:08.067926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 648/1309] 10.222.0.33 () {54 vars in 964 bytes} [Thu Oct 6 04:24:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:08.074423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be539a-0f41-426a-9cd5-b4d2cd4787f0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:08.074780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be539a-0f41-426a-9cd5-b4d2cd4787f0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:08.075308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be539a-0f41-426a-9cd5-b4d2cd4787f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:08.158215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f0be539a-0f41-426a-9cd5-b4d2cd4787f0 None None] MFA Rules not processed for user `87829678318948f3b81bd107076bdfa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:08.159560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f0be539a-0f41-426a-9cd5-b4d2cd4787f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:08.181716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:08.245413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 662/1310] 10.222.0.33 () {58 vars in 1061 bytes} [Thu Oct 6 04:24:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:08.254737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8664dbcc-1c10-4e36-820b-98e9746d3e11 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:08.256612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8664dbcc-1c10-4e36-820b-98e9746d3e11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:08.370511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8664dbcc-1c10-4e36-820b-98e9746d3e11 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:08.447586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa218b6a-8b8c-4002-8567-35219ae10444 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:08.448189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa218b6a-8b8c-4002-8567-35219ae10444 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:08.448460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa218b6a-8b8c-4002-8567-35219ae10444 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:08.448730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa218b6a-8b8c-4002-8567-35219ae10444 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:08.450428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fa218b6a-8b8c-4002-8567-35219ae10444 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:08.509079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:08.569052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa218b6a-8b8c-4002-8567-35219ae10444 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:08.570516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa218b6a-8b8c-4002-8567-35219ae10444 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:08.591086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 649/1311] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:24:11.014781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ad416-b66f-4ff2-a6be-1241ea641ab1 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:11.015327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ad416-b66f-4ff2-a6be-1241ea641ab1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:11.015327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ad416-b66f-4ff2-a6be-1241ea641ab1 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:11.017138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 663/1312] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:11.022474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a757aa9-26e0-4fa2-b4fb-0fcd727ece9f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:11.022887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a757aa9-26e0-4fa2-b4fb-0fcd727ece9f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:11.023098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a757aa9-26e0-4fa2-b4fb-0fcd727ece9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:11.094780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3a757aa9-26e0-4fa2-b4fb-0fcd727ece9f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:11.095999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3a757aa9-26e0-4fa2-b4fb-0fcd727ece9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:11.191876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 650/1313] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:11.473160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ad416-b66f-4ff2-a6be-1241ea641ab1 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:11.475987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1e5ad416-b66f-4ff2-a6be-1241ea641ab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:11.592935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ad416-b66f-4ff2-a6be-1241ea641ab1 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:11.670947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:11.673682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a65c22-d401-4bb8-bc3f-33e846cfd5c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:11.674726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a65c22-d401-4bb8-bc3f-33e846cfd5c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:11.675208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a65c22-d401-4bb8-bc3f-33e846cfd5c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:11.675677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a65c22-d401-4bb8-bc3f-33e846cfd5c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:11.677332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f9a65c22-d401-4bb8-bc3f-33e846cfd5c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:11.794723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a65c22-d401-4bb8-bc3f-33e846cfd5c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:11.796099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a65c22-d401-4bb8-bc3f-33e846cfd5c4 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:11.814994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 664/1314] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:14.253340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581fe9a-1c84-4e8c-9e8b-02e61295c540 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:14.253849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581fe9a-1c84-4e8c-9e8b-02e61295c540 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:14.253941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581fe9a-1c84-4e8c-9e8b-02e61295c540 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:14.255309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 651/1315] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:14.260398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630fcf2-3980-471c-b532-2da75bd6522f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:14.260818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630fcf2-3980-471c-b532-2da75bd6522f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:14.260936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d630fcf2-3980-471c-b532-2da75bd6522f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:14.326376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d630fcf2-3980-471c-b532-2da75bd6522f None None] MFA Rules not processed for user `38e7ae862c514700ac27c1f36f7caf25`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:14.327602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d630fcf2-3980-471c-b532-2da75bd6522f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:14.379972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:14.423645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 665/1316] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:14 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:14.709497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7581fe9a-1c84-4e8c-9e8b-02e61295c540 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:14.711070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7581fe9a-1c84-4e8c-9e8b-02e61295c540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:14.721820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:14.814710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7581fe9a-1c84-4e8c-9e8b-02e61295c540 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:14.886696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cae644-f3fd-4ee7-98fa-426c0e3d3415 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:14.887248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cae644-f3fd-4ee7-98fa-426c0e3d3415 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:14.887517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cae644-f3fd-4ee7-98fa-426c0e3d3415 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:14.887779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cae644-f3fd-4ee7-98fa-426c0e3d3415 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:14.889339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-78cae644-f3fd-4ee7-98fa-426c0e3d3415 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:14.949740 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:14.996660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cae644-f3fd-4ee7-98fa-426c0e3d3415 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:14.998077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cae644-f3fd-4ee7-98fa-426c0e3d3415 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:15.015311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 652/1317] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:14 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:17.705028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c533e6-39be-47b0-95bd-9f33b3788686 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:17.705569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c533e6-39be-47b0-95bd-9f33b3788686 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:17.705569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c533e6-39be-47b0-95bd-9f33b3788686 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:17.706886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 666/1318] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:17.712035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f4b079-793b-4ed7-a3c4-e17d14795133 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:17.712409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f4b079-793b-4ed7-a3c4-e17d14795133 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:17.712577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f4b079-793b-4ed7-a3c4-e17d14795133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:17.778112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f9f4b079-793b-4ed7-a3c4-e17d14795133 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:17.779384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f9f4b079-793b-4ed7-a3c4-e17d14795133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:17.881821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 653/1319] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:17 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:18.167550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c533e6-39be-47b0-95bd-9f33b3788686 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:18.169127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-78c533e6-39be-47b0-95bd-9f33b3788686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:18.274516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c533e6-39be-47b0-95bd-9f33b3788686 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:18.351270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49ce952-ae83-471e-b4d6-8ab196261f58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:18.351870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ce952-ae83-471e-b4d6-8ab196261f58 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:18.352175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ce952-ae83-471e-b4d6-8ab196261f58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:18.352445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49ce952-ae83-471e-b4d6-8ab196261f58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:18.354067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c49ce952-ae83-471e-b4d6-8ab196261f58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:18.361803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:18.465584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49ce952-ae83-471e-b4d6-8ab196261f58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:18.466916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49ce952-ae83-471e-b4d6-8ab196261f58 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:18.485101 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 667/1320] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:18 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:21.306505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79039e02-5f0e-4e46-b269-ee0685db2170 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:21.307075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79039e02-5f0e-4e46-b269-ee0685db2170 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:21.307075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79039e02-5f0e-4e46-b269-ee0685db2170 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:21.308425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 654/1321] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:21.314726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09509d7d-1c45-45a9-8e32-7e88cfc769cf None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:21.315322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09509d7d-1c45-45a9-8e32-7e88cfc769cf None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:21.315322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09509d7d-1c45-45a9-8e32-7e88cfc769cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:21.388681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-09509d7d-1c45-45a9-8e32-7e88cfc769cf None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:21.389917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-09509d7d-1c45-45a9-8e32-7e88cfc769cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:21.467238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:21.514231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 668/1322] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:21.802364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79039e02-5f0e-4e46-b269-ee0685db2170 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:21.803925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-79039e02-5f0e-4e46-b269-ee0685db2170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:21.824166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:21.911652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79039e02-5f0e-4e46-b269-ee0685db2170 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:21.986147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62299f3b-1f0d-4ed2-895c-f9c2e91e7ef8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:21.986749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62299f3b-1f0d-4ed2-895c-f9c2e91e7ef8 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:21.987058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62299f3b-1f0d-4ed2-895c-f9c2e91e7ef8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:21.987330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62299f3b-1f0d-4ed2-895c-f9c2e91e7ef8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:21.989209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-62299f3b-1f0d-4ed2-895c-f9c2e91e7ef8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:22.061434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:22.099342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62299f3b-1f0d-4ed2-895c-f9c2e91e7ef8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:22.100789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62299f3b-1f0d-4ed2-895c-f9c2e91e7ef8 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:22.119557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 655/1323] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:24.372996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fc75f-0666-47b7-bb93-a60ee0edcfaa None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:24.373575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fc75f-0666-47b7-bb93-a60ee0edcfaa None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:24.373575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fc75f-0666-47b7-bb93-a60ee0edcfaa None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:24.374870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 669/1324] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:24.381170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c16b6-f167-4a41-9b30-80d6becf0d0c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:24.381604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c16b6-f167-4a41-9b30-80d6becf0d0c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:24.381848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c16b6-f167-4a41-9b30-80d6becf0d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:24.456378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d49c16b6-f167-4a41-9b30-80d6becf0d0c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:24.457570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d49c16b6-f167-4a41-9b30-80d6becf0d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:24.553671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 656/1325] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:24.840199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02fc75f-0666-47b7-bb93-a60ee0edcfaa None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:24.841845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b02fc75f-0666-47b7-bb93-a60ee0edcfaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:24.948959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02fc75f-0666-47b7-bb93-a60ee0edcfaa None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:25.032622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc62a0d-b0e2-4129-b093-0304345ebdac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:25.033995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc62a0d-b0e2-4129-b093-0304345ebdac service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:25.034507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc62a0d-b0e2-4129-b093-0304345ebdac service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:25.035069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc62a0d-b0e2-4129-b093-0304345ebdac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:25.036987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3dc62a0d-b0e2-4129-b093-0304345ebdac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:25.057218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:25.154739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc62a0d-b0e2-4129-b093-0304345ebdac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:25.156978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc62a0d-b0e2-4129-b093-0304345ebdac service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:25.175665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 670/1326] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:27.377949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3cbfb8-b613-4f0b-a939-55e68db3a99b None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:27.378509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3cbfb8-b613-4f0b-a939-55e68db3a99b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:27.378509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3cbfb8-b613-4f0b-a939-55e68db3a99b None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:27.379873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 657/1327] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:27.385305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319c0553-d097-4081-a897-439ae1e2c1b2 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:27.385767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319c0553-d097-4081-a897-439ae1e2c1b2 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:27.385871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319c0553-d097-4081-a897-439ae1e2c1b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:27.452017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-319c0553-d097-4081-a897-439ae1e2c1b2 None None] MFA Rules not processed for user `38e7ae862c514700ac27c1f36f7caf25`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:27.453735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-319c0553-d097-4081-a897-439ae1e2c1b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:27.521458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:27.553860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 671/1328] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:27 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:27.848032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3cbfb8-b613-4f0b-a939-55e68db3a99b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:27.849565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5e3cbfb8-b613-4f0b-a939-55e68db3a99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:27.884122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:27.960675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3cbfb8-b613-4f0b-a939-55e68db3a99b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:28.035188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94d7ef7-e5d7-41ad-afde-9f11c2ee68e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:28.035791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94d7ef7-e5d7-41ad-afde-9f11c2ee68e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:28.036064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94d7ef7-e5d7-41ad-afde-9f11c2ee68e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:28.036353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94d7ef7-e5d7-41ad-afde-9f11c2ee68e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:28.037970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a94d7ef7-e5d7-41ad-afde-9f11c2ee68e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:28.115993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:28.148870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94d7ef7-e5d7-41ad-afde-9f11c2ee68e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:28.150316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94d7ef7-e5d7-41ad-afde-9f11c2ee68e6 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:28.168900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 658/1329] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:27 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:30.554878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d5f7a-3b92-4826-8f5d-4cb3267d6e32 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:30.555437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d5f7a-3b92-4826-8f5d-4cb3267d6e32 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:30.555437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45d5f7a-3b92-4826-8f5d-4cb3267d6e32 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:30.556686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 672/1330] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:30.561799 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35af079-d567-4b94-aba4-03b230e3690d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:30.562186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35af079-d567-4b94-aba4-03b230e3690d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:30.562905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35af079-d567-4b94-aba4-03b230e3690d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:30.628175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f35af079-d567-4b94-aba4-03b230e3690d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:30.629378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f35af079-d567-4b94-aba4-03b230e3690d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:30.728317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 659/1331] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:31.020400 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45d5f7a-3b92-4826-8f5d-4cb3267d6e32 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:31.021951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d45d5f7a-3b92-4826-8f5d-4cb3267d6e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:31.126297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45d5f7a-3b92-4826-8f5d-4cb3267d6e32 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:31.198083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398b38f4-088c-40ba-bff1-8f9048ab2795 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:31.198676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398b38f4-088c-40ba-bff1-8f9048ab2795 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:31.198952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398b38f4-088c-40ba-bff1-8f9048ab2795 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:31.199244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398b38f4-088c-40ba-bff1-8f9048ab2795 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:31.200833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-398b38f4-088c-40ba-bff1-8f9048ab2795 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:31.231066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:31.313116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398b38f4-088c-40ba-bff1-8f9048ab2795 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:31.314522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398b38f4-088c-40ba-bff1-8f9048ab2795 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:31.332977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 673/1332] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:33.522313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922b4400-bba0-4d17-9e61-ecb808db6ddb None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:33.522829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922b4400-bba0-4d17-9e61-ecb808db6ddb None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:33.522829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922b4400-bba0-4d17-9e61-ecb808db6ddb None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:33.524172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 660/1333] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:33.529292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdbac37-6597-4825-8d48-b3fa5ce78e67 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:33.529722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdbac37-6597-4825-8d48-b3fa5ce78e67 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:33.529815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdbac37-6597-4825-8d48-b3fa5ce78e67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:33.594150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-1fdbac37-6597-4825-8d48-b3fa5ce78e67 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:33.595370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1fdbac37-6597-4825-8d48-b3fa5ce78e67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:33.668746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:33.691680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 674/1334] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:33.995726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922b4400-bba0-4d17-9e61-ecb808db6ddb None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:33.997250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-922b4400-bba0-4d17-9e61-ecb808db6ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:34.038180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:34.104373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922b4400-bba0-4d17-9e61-ecb808db6ddb None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:34.199873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a888a186-0d59-4729-99aa-da9fec98a4db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:34.200475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888a186-0d59-4729-99aa-da9fec98a4db service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:34.200749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888a186-0d59-4729-99aa-da9fec98a4db service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:34.201018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888a186-0d59-4729-99aa-da9fec98a4db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:34.202628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a888a186-0d59-4729-99aa-da9fec98a4db service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:34.285405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:34.311475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a888a186-0d59-4729-99aa-da9fec98a4db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:34.313659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a888a186-0d59-4729-99aa-da9fec98a4db service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:34.331832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 661/1335] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:36.822268 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf7170c-a97c-40ae-a41a-fc9e5f35f1d0 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:36.822918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf7170c-a97c-40ae-a41a-fc9e5f35f1d0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:36.822918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf7170c-a97c-40ae-a41a-fc9e5f35f1d0 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:36.824188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 675/1336] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:36.830466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebc6b84-7bcd-4bf2-948d-145834790cf9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:36.831026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebc6b84-7bcd-4bf2-948d-145834790cf9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:36.831026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebc6b84-7bcd-4bf2-948d-145834790cf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:36.900509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6ebc6b84-7bcd-4bf2-948d-145834790cf9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:36.901628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6ebc6b84-7bcd-4bf2-948d-145834790cf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:37.004547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 662/1337] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:36 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:37.309096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ecfcd6-d1b2-42f7-9f4c-49fb6a1cd18e None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:37.309412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ecfcd6-d1b2-42f7-9f4c-49fb6a1cd18e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:37.309627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ecfcd6-d1b2-42f7-9f4c-49fb6a1cd18e None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:37.310979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 676/1338] 10.222.0.33 () {54 vars in 964 bytes} [Thu Oct 6 04:24:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:24:37.317781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a17b8-7ae6-4f38-8b06-d3a66dd72027 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:37.318063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a17b8-7ae6-4f38-8b06-d3a66dd72027 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:37.318333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a17b8-7ae6-4f38-8b06-d3a66dd72027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:37.372174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:37.394481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fa1a17b8-7ae6-4f38-8b06-d3a66dd72027 None None] MFA Rules not processed for user `87829678318948f3b81bd107076bdfa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:37.395716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fa1a17b8-7ae6-4f38-8b06-d3a66dd72027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:37.470919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 663/1339] 10.222.0.33 () {58 vars in 1061 bytes} [Thu Oct 6 04:24:37 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:37.479702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ecfcd6-d1b2-42f7-9f4c-49fb6a1cd18e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:37.481211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a8ecfcd6-d1b2-42f7-9f4c-49fb6a1cd18e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:37.583109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ecfcd6-d1b2-42f7-9f4c-49fb6a1cd18e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:37.657364 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833c2c7a-7546-4e53-9a1f-354336ca6e39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:37.657774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c2c7a-7546-4e53-9a1f-354336ca6e39 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:37.658046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c2c7a-7546-4e53-9a1f-354336ca6e39 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:37.658314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c2c7a-7546-4e53-9a1f-354336ca6e39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:37.659958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-833c2c7a-7546-4e53-9a1f-354336ca6e39 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:37.695663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:37.768935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c2c7a-7546-4e53-9a1f-354336ca6e39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:37.770288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c2c7a-7546-4e53-9a1f-354336ca6e39 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:37.787786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 677/1340] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:37 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:42.468420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c10b29-36fb-4ba5-a781-e88ad82c31d3 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:42.468954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c10b29-36fb-4ba5-a781-e88ad82c31d3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:42.468954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c10b29-36fb-4ba5-a781-e88ad82c31d3 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:42.470287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 664/1341] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:42.475694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714436bf-e2d8-4603-b18a-483339cb9931 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:42.476129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714436bf-e2d8-4603-b18a-483339cb9931 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:42.476368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714436bf-e2d8-4603-b18a-483339cb9931 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:42.544359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-714436bf-e2d8-4603-b18a-483339cb9931 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:42.545463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-714436bf-e2d8-4603-b18a-483339cb9931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:42.631479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:42.643377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 678/1342] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:42.924906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c10b29-36fb-4ba5-a781-e88ad82c31d3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:42.926487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c7c10b29-36fb-4ba5-a781-e88ad82c31d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:43.032425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c10b29-36fb-4ba5-a781-e88ad82c31d3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:43.070124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:43.105928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3498be5-c626-41c9-981d-f7d2f3ca4e47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:43.106529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3498be5-c626-41c9-981d-f7d2f3ca4e47 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:43.106804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3498be5-c626-41c9-981d-f7d2f3ca4e47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:43.107631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3498be5-c626-41c9-981d-f7d2f3ca4e47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:43.108768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e3498be5-c626-41c9-981d-f7d2f3ca4e47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:43.219182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3498be5-c626-41c9-981d-f7d2f3ca4e47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:43.220633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3498be5-c626-41c9-981d-f7d2f3ca4e47 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:43.241106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 665/1343] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:45.994793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c303fa-87b6-4fbd-b19b-0a635bfd9193 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:45.995327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c303fa-87b6-4fbd-b19b-0a635bfd9193 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:45.995327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c303fa-87b6-4fbd-b19b-0a635bfd9193 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:45.996613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 679/1344] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:46.001856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b36773-cbde-4e6d-aa83-8dc6170a3bd5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:46.002318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b36773-cbde-4e6d-aa83-8dc6170a3bd5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:46.002546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b36773-cbde-4e6d-aa83-8dc6170a3bd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:46.067804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-87b36773-cbde-4e6d-aa83-8dc6170a3bd5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:46.069005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-87b36773-cbde-4e6d-aa83-8dc6170a3bd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:46.086472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:46.169364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 666/1345] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:46 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:46.451358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c303fa-87b6-4fbd-b19b-0a635bfd9193 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:46.453405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-65c303fa-87b6-4fbd-b19b-0a635bfd9193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:46.561317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c303fa-87b6-4fbd-b19b-0a635bfd9193 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:46.643568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37e26b3-446c-4ee2-a9ff-bca4619a7245 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:46.644169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e26b3-446c-4ee2-a9ff-bca4619a7245 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:46.644493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e26b3-446c-4ee2-a9ff-bca4619a7245 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:46.644695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e26b3-446c-4ee2-a9ff-bca4619a7245 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:46.646294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c37e26b3-446c-4ee2-a9ff-bca4619a7245 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:46.690671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:46.757665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37e26b3-446c-4ee2-a9ff-bca4619a7245 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:46.759019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37e26b3-446c-4ee2-a9ff-bca4619a7245 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:46.778499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 680/1346] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:46 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:51.331920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0861fee4-9e7c-4949-a7c7-43973c4ea8f1 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:51.332359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0861fee4-9e7c-4949-a7c7-43973c4ea8f1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:51.332460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0861fee4-9e7c-4949-a7c7-43973c4ea8f1 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:51.334149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 667/1347] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:24:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:51.339543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd97866-0f38-4d16-8501-88f7b9ad70f9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:51.340036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd97866-0f38-4d16-8501-88f7b9ad70f9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:51.340146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd97866-0f38-4d16-8501-88f7b9ad70f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:51.408036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-dfd97866-0f38-4d16-8501-88f7b9ad70f9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:51.409232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dfd97866-0f38-4d16-8501-88f7b9ad70f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:51.502165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:51.504483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 681/1348] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:24:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:24:51.792365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0861fee4-9e7c-4949-a7c7-43973c4ea8f1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:24:51.793880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0861fee4-9e7c-4949-a7c7-43973c4ea8f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:51.898303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0861fee4-9e7c-4949-a7c7-43973c4ea8f1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:24:51.949565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:51.976068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbecbd2-e9bb-4be2-acc0-a6ccb62692c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:24:51.976669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbecbd2-e9bb-4be2-acc0-a6ccb62692c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:51.976944 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbecbd2-e9bb-4be2-acc0-a6ccb62692c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:51.977214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbecbd2-e9bb-4be2-acc0-a6ccb62692c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:51.978802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2dbecbd2-e9bb-4be2-acc0-a6ccb62692c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:52.088346 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbecbd2-e9bb-4be2-acc0-a6ccb62692c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:24:52.089791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbecbd2-e9bb-4be2-acc0-a6ccb62692c3 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:24:52.107435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 668/1349] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:24:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:24:57.153435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c7de65-f728-4a2c-8e9d-99fd7682f341 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:57.153944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c7de65-f728-4a2c-8e9d-99fd7682f341 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:57.153944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c7de65-f728-4a2c-8e9d-99fd7682f341 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:57.156180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 682/1350] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:24:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:24:57.161294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c0126-dd91-4423-8709-71535faee516 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:24:57.161755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c0126-dd91-4423-8709-71535faee516 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:24:57.161853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c0126-dd91-4423-8709-71535faee516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:24:57.260151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-059c0126-dd91-4423-8709-71535faee516 None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:24:57.261675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-059c0126-dd91-4423-8709-71535faee516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:24:57.282923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:24:57.434814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 669/1351] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:24:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:01.618729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a85c9c4-119a-4ad8-bc00-1b03103799e9 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:01.619366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a85c9c4-119a-4ad8-bc00-1b03103799e9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:01.619366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a85c9c4-119a-4ad8-bc00-1b03103799e9 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:01.620815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 683/1352] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:25:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:01.626263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35fdd00-5ffc-4bd2-9bf4-3a1fc643897d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:01.626748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35fdd00-5ffc-4bd2-9bf4-3a1fc643897d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:01.626833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35fdd00-5ffc-4bd2-9bf4-3a1fc643897d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:01.707780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f35fdd00-5ffc-4bd2-9bf4-3a1fc643897d None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:01.709251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f35fdd00-5ffc-4bd2-9bf4-3a1fc643897d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:01.784801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:01.822671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 670/1353] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:25:01 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:03.491456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c761b530-5287-4616-bec2-61d2c82ed93e None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:03.491973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c761b530-5287-4616-bec2-61d2c82ed93e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:03.491973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c761b530-5287-4616-bec2-61d2c82ed93e None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:03.493382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 684/1354] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:25:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:03.498632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e71a635-3df1-41b0-b32a-4745c3c385e9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:03.499224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e71a635-3df1-41b0-b32a-4745c3c385e9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:03.499224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e71a635-3df1-41b0-b32a-4745c3c385e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:03.582024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5e71a635-3df1-41b0-b32a-4745c3c385e9 None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:03.583317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5e71a635-3df1-41b0-b32a-4745c3c385e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:03.706402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 671/1355] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:25:03 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:25:07.072910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d95bc-7ef0-4718-a082-9e83786c2e56 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:07.073477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d95bc-7ef0-4718-a082-9e83786c2e56 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:07.073477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d95bc-7ef0-4718-a082-9e83786c2e56 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:07.074766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 685/1356] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:07.080593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba7484b-2256-4863-8780-20ee4888d0e2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:07.081087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba7484b-2256-4863-8780-20ee4888d0e2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:07.081178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba7484b-2256-4863-8780-20ee4888d0e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:07.121025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:07.160041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-0ba7484b-2256-4863-8780-20ee4888d0e2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:07.161261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0ba7484b-2256-4863-8780-20ee4888d0e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:07.268921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 672/1357] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:25:07.607905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebc95f3-fa85-4285-beba-05b9182d01a3 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:07.608617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebc95f3-fa85-4285-beba-05b9182d01a3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:07.609109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebc95f3-fa85-4285-beba-05b9182d01a3 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:07.611021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 686/1358] 10.222.0.33 () {54 vars in 961 bytes} [Thu Oct 6 04:25:07 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:07.618536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0a050a-b150-462c-b326-0dcf60d79915 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:07.618827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0a050a-b150-462c-b326-0dcf60d79915 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:07.619133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0a050a-b150-462c-b326-0dcf60d79915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:07.725997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-0c0a050a-b150-462c-b326-0dcf60d79915 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:07.727361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0c0a050a-b150-462c-b326-0dcf60d79915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:07.787001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:07.859190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 673/1359] 10.222.0.33 () {58 vars in 1058 bytes} [Thu Oct 6 04:25:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:07.867306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebc95f3-fa85-4285-beba-05b9182d01a3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:07.869420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9ebc95f3-fa85-4285-beba-05b9182d01a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:08.023892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebc95f3-fa85-4285-beba-05b9182d01a3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:08.102032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:08.157359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98419ed-8979-40f9-9601-ab881413c1a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:08.158059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98419ed-8979-40f9-9601-ab881413c1a9 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:08.158436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98419ed-8979-40f9-9601-ab881413c1a9 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:08.158802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98419ed-8979-40f9-9601-ab881413c1a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:08.160528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f98419ed-8979-40f9-9601-ab881413c1a9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:08.267831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98419ed-8979-40f9-9601-ab881413c1a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:08.269608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98419ed-8979-40f9-9601-ab881413c1a9 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:08.288947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 687/1360] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:10.692197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd36c6-5815-4801-a3c8-eb5d43077c83 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:10.692197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd36c6-5815-4801-a3c8-eb5d43077c83 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:10.692197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd36c6-5815-4801-a3c8-eb5d43077c83 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:10.697035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 674/1361] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:10.699313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc3b583-b358-4960-8e02-e3cd8c010c12 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:10.699313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc3b583-b358-4960-8e02-e3cd8c010c12 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:10.699313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc3b583-b358-4960-8e02-e3cd8c010c12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:10.772644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bcc3b583-b358-4960-8e02-e3cd8c010c12 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:10.773863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bcc3b583-b358-4960-8e02-e3cd8c010c12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:10.814160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:10.883005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 688/1362] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:25:11.167939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fd36c6-5815-4801-a3c8-eb5d43077c83 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:11.169453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f0fd36c6-5815-4801-a3c8-eb5d43077c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:11.344593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fd36c6-5815-4801-a3c8-eb5d43077c83 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:11.356077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:11.493366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc5ddd0-91e8-4320-b001-aa194a2545c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:11.494081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc5ddd0-91e8-4320-b001-aa194a2545c0 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:11.494368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc5ddd0-91e8-4320-b001-aa194a2545c0 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:11.494633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc5ddd0-91e8-4320-b001-aa194a2545c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:11.496265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9cc5ddd0-91e8-4320-b001-aa194a2545c0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:11.591520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc5ddd0-91e8-4320-b001-aa194a2545c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:11.593670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc5ddd0-91e8-4320-b001-aa194a2545c0 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:11.612435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 675/1363] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:14.101859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db8b608-8d6d-459b-bfd7-c6f1e8b5b530 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:14.102397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db8b608-8d6d-459b-bfd7-c6f1e8b5b530 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:14.102397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db8b608-8d6d-459b-bfd7-c6f1e8b5b530 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:14.104045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 689/1364] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:14.109817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60048a-9f93-4828-ad95-b6406cd21b5f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:14.110542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60048a-9f93-4828-ad95-b6406cd21b5f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:14.110635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce60048a-9f93-4828-ad95-b6406cd21b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:14.127312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:14.183149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ce60048a-9f93-4828-ad95-b6406cd21b5f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:14.184270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ce60048a-9f93-4828-ad95-b6406cd21b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:14.294994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 676/1365] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:14.589237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2794f-c9c3-467c-815f-e37acc944f18 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:14.589842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2794f-c9c3-467c-815f-e37acc944f18 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:14.590312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed2794f-c9c3-467c-815f-e37acc944f18 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:14.592904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 690/1366] 10.222.0.33 () {54 vars in 961 bytes} [Thu Oct 6 04:25:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:14.597752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19660f5e-48e6-49a6-8e35-92affe1bc827 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:14.598038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19660f5e-48e6-49a6-8e35-92affe1bc827 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:14.598310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19660f5e-48e6-49a6-8e35-92affe1bc827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:14.678833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-19660f5e-48e6-49a6-8e35-92affe1bc827 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:14.680075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-19660f5e-48e6-49a6-8e35-92affe1bc827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:14.715776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:14.802582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 677/1367] 10.222.0.33 () {58 vars in 1058 bytes} [Thu Oct 6 04:25:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:14.810339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed2794f-c9c3-467c-815f-e37acc944f18 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:14.812445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5ed2794f-c9c3-467c-815f-e37acc944f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:14.979811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed2794f-c9c3-467c-815f-e37acc944f18 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:15.003838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:15.105037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf5419c-a154-4f73-8d58-ea42e4cd1135 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:15.105722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf5419c-a154-4f73-8d58-ea42e4cd1135 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:15.105956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf5419c-a154-4f73-8d58-ea42e4cd1135 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:15.106155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf5419c-a154-4f73-8d58-ea42e4cd1135 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:15.107819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fcf5419c-a154-4f73-8d58-ea42e4cd1135 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:15.213357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf5419c-a154-4f73-8d58-ea42e4cd1135 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:15.215118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf5419c-a154-4f73-8d58-ea42e4cd1135 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:15.241295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 691/1368] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:15.282559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865d63f0-b100-4373-b5e5-b14c1d7f844f None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:15.283148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865d63f0-b100-4373-b5e5-b14c1d7f844f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:15.283275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865d63f0-b100-4373-b5e5-b14c1d7f844f None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:15.284900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 678/1369] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:15.291172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c239ed-8b8e-4a18-a5fb-0a4b9610c242 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:15.291788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c239ed-8b8e-4a18-a5fb-0a4b9610c242 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:15.292318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c239ed-8b8e-4a18-a5fb-0a4b9610c242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:15.329245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:15.374438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-60c239ed-8b8e-4a18-a5fb-0a4b9610c242 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:15.376046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-60c239ed-8b8e-4a18-a5fb-0a4b9610c242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:15.498327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 692/1370] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:15.798423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba5fdb6-3432-4496-b33a-147329fffb31 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:15.798709 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba5fdb6-3432-4496-b33a-147329fffb31 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:15.799008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba5fdb6-3432-4496-b33a-147329fffb31 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:15.800489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 679/1371] 10.222.0.33 () {54 vars in 963 bytes} [Thu Oct 6 04:25:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 06 04:25:15.805612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d503c46-6b3d-4107-938e-c2e518c95f8e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:15.805895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d503c46-6b3d-4107-938e-c2e518c95f8e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:15.806185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d503c46-6b3d-4107-938e-c2e518c95f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:15.883586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3d503c46-6b3d-4107-938e-c2e518c95f8e None None] MFA Rules not processed for user `61d1b78e81db4dbe987d788fd398e89d`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:15.884772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3d503c46-6b3d-4107-938e-c2e518c95f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:15.934095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:15.967447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 693/1372] 10.222.0.33 () {58 vars in 1060 bytes} [Thu Oct 6 04:25:15 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:15.975222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba5fdb6-3432-4496-b33a-147329fffb31 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:15.976758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8ba5fdb6-3432-4496-b33a-147329fffb31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:16.097822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba5fdb6-3432-4496-b33a-147329fffb31 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:16.142803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:16.182113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5299cc-f398-4ae2-b917-57858621a4ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:16.182728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5299cc-f398-4ae2-b917-57858621a4ce service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:16.183030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5299cc-f398-4ae2-b917-57858621a4ce service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:16.183305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5299cc-f398-4ae2-b917-57858621a4ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:16.184919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3e5299cc-f398-4ae2-b917-57858621a4ce service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:16.331545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5299cc-f398-4ae2-b917-57858621a4ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:16.333292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5299cc-f398-4ae2-b917-57858621a4ce service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:16.352497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 680/1373] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:17.473061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c560471-5e44-459f-9edf-6be5977db1b8 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:17.473639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c560471-5e44-459f-9edf-6be5977db1b8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:17.473639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c560471-5e44-459f-9edf-6be5977db1b8 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:17.474977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 694/1374] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:17.480299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad57d7b5-f3b3-4fa0-a26f-b87d867b3a0f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:17.480741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad57d7b5-f3b3-4fa0-a26f-b87d867b3a0f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:17.480970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad57d7b5-f3b3-4fa0-a26f-b87d867b3a0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:17.548682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ad57d7b5-f3b3-4fa0-a26f-b87d867b3a0f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:17.549888 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ad57d7b5-f3b3-4fa0-a26f-b87d867b3a0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:17.571449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:17.648541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 681/1375] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:17.954710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c560471-5e44-459f-9edf-6be5977db1b8 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:17.956358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5c560471-5e44-459f-9edf-6be5977db1b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:18.119568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c560471-5e44-459f-9edf-6be5977db1b8 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:18.175207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:18.250061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33124494-b9bb-4bd1-b4db-7af8fc73de7f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:18.250678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33124494-b9bb-4bd1-b4db-7af8fc73de7f service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:18.250977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33124494-b9bb-4bd1-b4db-7af8fc73de7f service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:18.251258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33124494-b9bb-4bd1-b4db-7af8fc73de7f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:18.252876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-33124494-b9bb-4bd1-b4db-7af8fc73de7f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:18.349346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33124494-b9bb-4bd1-b4db-7af8fc73de7f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:18.351067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33124494-b9bb-4bd1-b4db-7af8fc73de7f service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:18.370847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 695/1376] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:20.659377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3223d2a4-2328-4b66-94cd-96a8f4d83885 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:20.659940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3223d2a4-2328-4b66-94cd-96a8f4d83885 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:20.659940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3223d2a4-2328-4b66-94cd-96a8f4d83885 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:20.662344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 682/1377] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:20.666571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91babdfb-f748-4759-9a1b-e7e80ef66fa8 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:20.666982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91babdfb-f748-4759-9a1b-e7e80ef66fa8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:20.667159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91babdfb-f748-4759-9a1b-e7e80ef66fa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:20.742709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-91babdfb-f748-4759-9a1b-e7e80ef66fa8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:20.743982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-91babdfb-f748-4759-9a1b-e7e80ef66fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:20.753195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:20.864511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 696/1378] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:25:21.153495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3223d2a4-2328-4b66-94cd-96a8f4d83885 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:21.155709 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3223d2a4-2328-4b66-94cd-96a8f4d83885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:21.307243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3223d2a4-2328-4b66-94cd-96a8f4d83885 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:21.310811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:21.434305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54b2e96-79f9-4243-a8cc-b09d6f86ec57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:21.434828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54b2e96-79f9-4243-a8cc-b09d6f86ec57 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:21.435304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54b2e96-79f9-4243-a8cc-b09d6f86ec57 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:21.435634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54b2e96-79f9-4243-a8cc-b09d6f86ec57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:21.437343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c54b2e96-79f9-4243-a8cc-b09d6f86ec57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:21.538332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54b2e96-79f9-4243-a8cc-b09d6f86ec57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:21.539977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54b2e96-79f9-4243-a8cc-b09d6f86ec57 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:21.561823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 683/1379] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:25:23.784999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69be633-1fb5-4dc8-8775-53dd04d70060 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:23.785500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69be633-1fb5-4dc8-8775-53dd04d70060 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:23.785686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69be633-1fb5-4dc8-8775-53dd04d70060 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:23.787033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 697/1380] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:23.791981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac004012-e468-4da7-a2ad-ba29646ca091 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:23.792363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac004012-e468-4da7-a2ad-ba29646ca091 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:23.792557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac004012-e468-4da7-a2ad-ba29646ca091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:23.804898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:23.863523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ac004012-e468-4da7-a2ad-ba29646ca091 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:23.864455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ac004012-e468-4da7-a2ad-ba29646ca091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:23.977104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 684/1381] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:24.283847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69be633-1fb5-4dc8-8775-53dd04d70060 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:24.285449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e69be633-1fb5-4dc8-8775-53dd04d70060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:24.423794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:24.434575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69be633-1fb5-4dc8-8775-53dd04d70060 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:24.555099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b81e88-46b2-4b17-939c-aaa8dcfeb853 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:24.555775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b81e88-46b2-4b17-939c-aaa8dcfeb853 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:24.556173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b81e88-46b2-4b17-939c-aaa8dcfeb853 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:24.556508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b81e88-46b2-4b17-939c-aaa8dcfeb853 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:24.558552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-96b81e88-46b2-4b17-939c-aaa8dcfeb853 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:24.661475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b81e88-46b2-4b17-939c-aaa8dcfeb853 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:24.663115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b81e88-46b2-4b17-939c-aaa8dcfeb853 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:24.674721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:24.683951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 698/1382] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:27.172496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1c0d50-a63e-4e69-881a-e78507e1c276 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:27.173031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1c0d50-a63e-4e69-881a-e78507e1c276 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:27.173031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1c0d50-a63e-4e69-881a-e78507e1c276 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:27.174552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 685/1383] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:27.180460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32803f-dc89-4ec8-b744-ed5036ffa854 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:27.180880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32803f-dc89-4ec8-b744-ed5036ffa854 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:27.181225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa32803f-dc89-4ec8-b744-ed5036ffa854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:27.261003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-fa32803f-dc89-4ec8-b744-ed5036ffa854 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:27.262213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fa32803f-dc89-4ec8-b744-ed5036ffa854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:27.365149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 699/1384] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:27.677062 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1c0d50-a63e-4e69-881a-e78507e1c276 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:27.679102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1b1c0d50-a63e-4e69-881a-e78507e1c276 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:27.778361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:27.845814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1c0d50-a63e-4e69-881a-e78507e1c276 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:27.971980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805c789d-137b-4daa-b580-5fa945619a0e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:27.972548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805c789d-137b-4daa-b580-5fa945619a0e service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:27.972846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805c789d-137b-4daa-b580-5fa945619a0e service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:27.973141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805c789d-137b-4daa-b580-5fa945619a0e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:27.974852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-805c789d-137b-4daa-b580-5fa945619a0e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:28.029998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:28.079612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805c789d-137b-4daa-b580-5fa945619a0e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:28.081672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805c789d-137b-4daa-b580-5fa945619a0e service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:28.104858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 686/1385] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:30.344054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe3eaaa-10d6-416a-b378-a4f72bc00f59 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:30.344606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe3eaaa-10d6-416a-b378-a4f72bc00f59 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:30.344606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe3eaaa-10d6-416a-b378-a4f72bc00f59 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:30.345932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 700/1386] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:30.352022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16eb95-38f3-4e34-b984-188f50af629b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:30.352461 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16eb95-38f3-4e34-b984-188f50af629b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:30.352724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16eb95-38f3-4e34-b984-188f50af629b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:30.427974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-bb16eb95-38f3-4e34-b984-188f50af629b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:30.429475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-bb16eb95-38f3-4e34-b984-188f50af629b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:30.538179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 687/1387] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:30.851487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe3eaaa-10d6-416a-b378-a4f72bc00f59 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:30.852978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1fe3eaaa-10d6-416a-b378-a4f72bc00f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:30.929433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:31.000017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe3eaaa-10d6-416a-b378-a4f72bc00f59 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:31.113550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad55a62c-3b65-4666-86f9-0a5179571ff7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:31.114060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55a62c-3b65-4666-86f9-0a5179571ff7 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:31.114335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55a62c-3b65-4666-86f9-0a5179571ff7 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:31.114603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55a62c-3b65-4666-86f9-0a5179571ff7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:31.116243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ad55a62c-3b65-4666-86f9-0a5179571ff7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:31.156103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:31.208790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad55a62c-3b65-4666-86f9-0a5179571ff7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:31.210985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad55a62c-3b65-4666-86f9-0a5179571ff7 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:31.230189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 701/1388] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:32.834159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d1331-bd5c-4c23-ae42-8c30ce765317 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:32.834719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d1331-bd5c-4c23-ae42-8c30ce765317 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:32.834719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0d1331-bd5c-4c23-ae42-8c30ce765317 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:32.836082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 688/1389] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:32.841550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a93721-964b-4546-b1bf-e760c7e0e3ba None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:32.841990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a93721-964b-4546-b1bf-e760c7e0e3ba None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:32.842171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a93721-964b-4546-b1bf-e760c7e0e3ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:32.915254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d0a93721-964b-4546-b1bf-e760c7e0e3ba None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:32.916463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d0a93721-964b-4546-b1bf-e760c7e0e3ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:33.031522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 702/1390] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:33.344999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a46fd75-ec17-411e-b757-a84a14469c08 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:33.345282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a46fd75-ec17-411e-b757-a84a14469c08 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:33.345555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a46fd75-ec17-411e-b757-a84a14469c08 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:33.346987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 689/1391] 10.222.0.33 () {54 vars in 962 bytes} [Thu Oct 6 04:25:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:33.352287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2668d-f24c-452c-85b9-f20941368b68 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:33.352580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2668d-f24c-452c-85b9-f20941368b68 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:33.352982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2668d-f24c-452c-85b9-f20941368b68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:33.369662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:33.413668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca3eb6-ea66-4ac9-b693-7d39264749b9 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:33.413978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca3eb6-ea66-4ac9-b693-7d39264749b9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:33.414925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca3eb6-ea66-4ac9-b693-7d39264749b9 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:33.415748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 690/1392] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:33.421114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:33.421394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:33.421668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:33.437107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-44e2668d-f24c-452c-85b9-f20941368b68 None None] MFA Rules not processed for user `b34c61ebdb9147d5b3d66cbfa556f38e`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:33.438314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-44e2668d-f24c-452c-85b9-f20941368b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:33.458112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:33.491678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:33.492922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:33.519400 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 703/1393] 10.222.0.33 () {58 vars in 1059 bytes} [Thu Oct 6 04:25:33 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:25:33.528334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e2668d-f24c-452c-85b9-f20941368b68 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:33.536201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e2668d-f24c-452c-85b9-f20941368b68 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:33.595631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 691/1394] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:25:33.611284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b5e604-75ea-49ce-aeb9-aa8c5bc2b61f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b34c61ebdb9147d5b3d66cbfa556f38e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:33.611877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5e604-75ea-49ce-aeb9-aa8c5bc2b61f service glance] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:33.612150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5e604-75ea-49ce-aeb9-aa8c5bc2b61f service glance] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:33.612415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b5e604-75ea-49ce-aeb9-aa8c5bc2b61f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:33.613999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c6b5e604-75ea-49ce-aeb9-aa8c5bc2b61f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:33.643972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:33.728513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b5e604-75ea-49ce-aeb9-aa8c5bc2b61f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:33.729926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b5e604-75ea-49ce-aeb9-aa8c5bc2b61f service glance] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:33.748945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 704/1395] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:25:33.891558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:33.892755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:34.013964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:34.021959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec27cab6-bea3-49b8-87d3-4e4b60566e0b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:34.148163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ca18bf-3e47-4d8b-bfca-c74f53ed2365 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:34.148956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ca18bf-3e47-4d8b-bfca-c74f53ed2365 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:34.149401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ca18bf-3e47-4d8b-bfca-c74f53ed2365 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:34.149672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ca18bf-3e47-4d8b-bfca-c74f53ed2365 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:34.151640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d6ca18bf-3e47-4d8b-bfca-c74f53ed2365 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:34.250275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ca18bf-3e47-4d8b-bfca-c74f53ed2365 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:34.251908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ca18bf-3e47-4d8b-bfca-c74f53ed2365 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:34.268022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:34.276289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 692/1396] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:34.542428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97aaad-5c9a-44d4-8f03-3e1104d15e98 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:34.542889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97aaad-5c9a-44d4-8f03-3e1104d15e98 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:34.543170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97aaad-5c9a-44d4-8f03-3e1104d15e98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:34.617162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0d97aaad-5c9a-44d4-8f03-3e1104d15e98 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:34.619739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0d97aaad-5c9a-44d4-8f03-3e1104d15e98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:34.709744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:34.716027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 705/1397] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:25:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:34.762013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e99379-2e94-467b-b316-6512d2ff38e6 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:34.762294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e99379-2e94-467b-b316-6512d2ff38e6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:34.762565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e99379-2e94-467b-b316-6512d2ff38e6 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:34.764555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 693/1398] 10.222.0.33 () {54 vars in 943 bytes} [Thu Oct 6 04:25:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:34.770829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497db840-e7bd-4208-b760-4af70c110434 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:34.771135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497db840-e7bd-4208-b760-4af70c110434 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:34.771402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497db840-e7bd-4208-b760-4af70c110434 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:34.847373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-497db840-e7bd-4208-b760-4af70c110434 None None] MFA Rules not processed for user `2e280c4b418e4aafb822a617424c7e11`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:34.848562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-497db840-e7bd-4208-b760-4af70c110434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:34.956514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 706/1399] 10.222.0.33 () {58 vars in 1040 bytes} [Thu Oct 6 04:25:34 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:25:34.971036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e99379-2e94-467b-b316-6512d2ff38e6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:34.973974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a4e99379-2e94-467b-b316-6512d2ff38e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:35.122437 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e99379-2e94-467b-b316-6512d2ff38e6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:35.208633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:35.233652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381b40d-e622-4ca6-abb8-1e85729af9dc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:35.234293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381b40d-e622-4ca6-abb8-1e85729af9dc service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:35.234569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381b40d-e622-4ca6-abb8-1e85729af9dc service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:35.235151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381b40d-e622-4ca6-abb8-1e85729af9dc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:35.237089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f381b40d-e622-4ca6-abb8-1e85729af9dc service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:35.323466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381b40d-e622-4ca6-abb8-1e85729af9dc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:35.325058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381b40d-e622-4ca6-abb8-1e85729af9dc service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:35.334204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 694/1400] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:25:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:25:36.520254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b4dc2d-f635-4744-9871-4582155ea7dc None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:36.521576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b4dc2d-f635-4744-9871-4582155ea7dc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:36.521973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b4dc2d-f635-4744-9871-4582155ea7dc None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:36.523580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 707/1401] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:36.531144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359ff8d-95be-437c-8df0-899ae5fd0327 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:36.531750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359ff8d-95be-437c-8df0-899ae5fd0327 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:36.532281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359ff8d-95be-437c-8df0-899ae5fd0327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:36.609344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1359ff8d-95be-437c-8df0-899ae5fd0327 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:36.611379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1359ff8d-95be-437c-8df0-899ae5fd0327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:36.696546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:36.730254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 695/1402] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:37.029470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b4dc2d-f635-4744-9871-4582155ea7dc None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:37.032439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-91b4dc2d-f635-4744-9871-4582155ea7dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:37.117490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:37.215929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b4dc2d-f635-4744-9871-4582155ea7dc None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:37.348126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df791f9-b106-4e96-a836-58c2e809317b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:37.348718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df791f9-b106-4e96-a836-58c2e809317b service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:37.349000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df791f9-b106-4e96-a836-58c2e809317b service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:37.349272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df791f9-b106-4e96-a836-58c2e809317b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:37.350875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3df791f9-b106-4e96-a836-58c2e809317b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:37.376672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:37.463902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df791f9-b106-4e96-a836-58c2e809317b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:37.465635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df791f9-b106-4e96-a836-58c2e809317b service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:37.486152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 708/1403] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:39.712428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bbf21-09e8-4516-bdaf-c89c72a94e87 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:39.713013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bbf21-09e8-4516-bdaf-c89c72a94e87 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:39.713013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bbf21-09e8-4516-bdaf-c89c72a94e87 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:39.714449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 696/1404] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:39.720034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc1a17-ebf7-472f-9a1a-056431cd697d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:39.720575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc1a17-ebf7-472f-9a1a-056431cd697d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:39.720575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc1a17-ebf7-472f-9a1a-056431cd697d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:39.789503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d7bc1a17-ebf7-472f-9a1a-056431cd697d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:39.791264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d7bc1a17-ebf7-472f-9a1a-056431cd697d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:39.893266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:39.901203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 709/1405] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:40.198152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bbf21-09e8-4516-bdaf-c89c72a94e87 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:40.199788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c63bbf21-09e8-4516-bdaf-c89c72a94e87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:40.376810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bbf21-09e8-4516-bdaf-c89c72a94e87 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:40.437278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:40.506733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5412a1-40e1-4744-9079-e3bbe42c6f17 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:40.507374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5412a1-40e1-4744-9079-e3bbe42c6f17 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:40.507651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5412a1-40e1-4744-9079-e3bbe42c6f17 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:40.507921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5412a1-40e1-4744-9079-e3bbe42c6f17 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:40.509651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9a5412a1-40e1-4744-9079-e3bbe42c6f17 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:40.613629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5412a1-40e1-4744-9079-e3bbe42c6f17 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:40.615310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5412a1-40e1-4744-9079-e3bbe42c6f17 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:40.635579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 697/1406] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:42.918936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7328695-d45f-4fc8-8b13-43f5835ff5ef None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:42.922406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7328695-d45f-4fc8-8b13-43f5835ff5ef None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:42.922406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7328695-d45f-4fc8-8b13-43f5835ff5ef None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:42.925704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 710/1407] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:25:42 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:42.927507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e349ca-6c2a-43aa-858d-5b9eaf14959f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:42.928296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e349ca-6c2a-43aa-858d-5b9eaf14959f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:42.928593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e349ca-6c2a-43aa-858d-5b9eaf14959f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:43.002012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c2e349ca-6c2a-43aa-858d-5b9eaf14959f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:43.003266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c2e349ca-6c2a-43aa-858d-5b9eaf14959f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:43.018542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:43.112295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 698/1408] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:25:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:43.405884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7328695-d45f-4fc8-8b13-43f5835ff5ef None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:25:43.407427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b7328695-d45f-4fc8-8b13-43f5835ff5ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:43.572922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7328695-d45f-4fc8-8b13-43f5835ff5ef None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:25:43.648187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:43.700807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eb289c-2bfd-4cd9-a088-0bf4ce1da957 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:25:43.701506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eb289c-2bfd-4cd9-a088-0bf4ce1da957 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:43.701785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eb289c-2bfd-4cd9-a088-0bf4ce1da957 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:43.702060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eb289c-2bfd-4cd9-a088-0bf4ce1da957 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:43.703708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a6eb289c-2bfd-4cd9-a088-0bf4ce1da957 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:43.803915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eb289c-2bfd-4cd9-a088-0bf4ce1da957 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:25:43.805520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eb289c-2bfd-4cd9-a088-0bf4ce1da957 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:25:43.827562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 711/1409] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:25:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:25:53.105441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=37 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:53.824324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=65 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:58.131986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d400c1e-3397-43e6-97c9-81becb115783 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:58.132652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d400c1e-3397-43e6-97c9-81becb115783 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:58.132652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d400c1e-3397-43e6-97c9-81becb115783 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:58.133883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 699/1410] 10.222.0.33 () {54 vars in 918 bytes} [Thu Oct 6 04:25:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:58.139866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ad9ed4-e117-4aac-829a-175489c1da8e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:58.140483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ad9ed4-e117-4aac-829a-175489c1da8e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:58.140483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ad9ed4-e117-4aac-829a-175489c1da8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:58.216495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-50ad9ed4-e117-4aac-829a-175489c1da8e None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:58.218320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-50ad9ed4-e117-4aac-829a-175489c1da8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:58.324862 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 712/1411] 10.222.0.33 () {58 vars in 1015 bytes} [Thu Oct 6 04:25:58 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:25:58.339352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900d8db-eb9e-4585-a40b-7c8962f10c72 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:58.339635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900d8db-eb9e-4585-a40b-7c8962f10c72 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:58.339905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900d8db-eb9e-4585-a40b-7c8962f10c72 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:58.341268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 700/1412] 10.222.0.33 () {54 vars in 918 bytes} [Thu Oct 6 04:25:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:25:58.346076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3bb272-a1fc-4154-b377-d40794856736 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:25:58.346361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3bb272-a1fc-4154-b377-d40794856736 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:25:58.346637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3bb272-a1fc-4154-b377-d40794856736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:25:58.422623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:25:58.425778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3f3bb272-a1fc-4154-b377-d40794856736 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:25:58.427565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3f3bb272-a1fc-4154-b377-d40794856736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:25:58.546069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 713/1413] 10.222.0.33 () {58 vars in 1015 bytes} [Thu Oct 6 04:25:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:01.129420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8df0bd-05a4-40c5-83c3-ca875fa71698 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:01.129986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8df0bd-05a4-40c5-83c3-ca875fa71698 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:01.129986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8df0bd-05a4-40c5-83c3-ca875fa71698 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:01.131416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 701/1414] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:01.136742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e77253-318f-4b69-b453-2efa13a58698 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:01.137144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e77253-318f-4b69-b453-2efa13a58698 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:01.137287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e77253-318f-4b69-b453-2efa13a58698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:01.203167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-51e77253-318f-4b69-b453-2efa13a58698 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:01.204363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-51e77253-318f-4b69-b453-2efa13a58698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:01.260734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:01.301976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 714/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:01.600671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8df0bd-05a4-40c5-83c3-ca875fa71698 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:01.602252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5c8df0bd-05a4-40c5-83c3-ca875fa71698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:01.746492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8df0bd-05a4-40c5-83c3-ca875fa71698 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:01.818689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:01.860940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359fae43-6970-4a5f-bb78-9d7e461b0963 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:01.861719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359fae43-6970-4a5f-bb78-9d7e461b0963 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:01.861996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359fae43-6970-4a5f-bb78-9d7e461b0963 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:01.862267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359fae43-6970-4a5f-bb78-9d7e461b0963 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:01.863911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-359fae43-6970-4a5f-bb78-9d7e461b0963 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:01.952431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359fae43-6970-4a5f-bb78-9d7e461b0963 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:01.954099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359fae43-6970-4a5f-bb78-9d7e461b0963 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:01.972301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 702/1416] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:03.116112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9359c4-89ea-4bcc-94dc-613e7aa0e3e0 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:03.116719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9359c4-89ea-4bcc-94dc-613e7aa0e3e0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:03.116719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9359c4-89ea-4bcc-94dc-613e7aa0e3e0 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:03.118037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 715/1417] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:03.124635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105e4159-fcf2-48ac-8865-bd0570c474f5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:03.125068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105e4159-fcf2-48ac-8865-bd0570c474f5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:03.125161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105e4159-fcf2-48ac-8865-bd0570c474f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:03.199989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-105e4159-fcf2-48ac-8865-bd0570c474f5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:03.201214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-105e4159-fcf2-48ac-8865-bd0570c474f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:03.228754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:03.299366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 703/1418] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:03.459737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9359c4-89ea-4bcc-94dc-613e7aa0e3e0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:03.461136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fa9359c4-89ea-4bcc-94dc-613e7aa0e3e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:03.605968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9359c4-89ea-4bcc-94dc-613e7aa0e3e0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:03.645417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:03.717842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd00606c-dc40-4513-840f-c8bfc3761c51 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:03.718359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd00606c-dc40-4513-840f-c8bfc3761c51 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:03.718635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd00606c-dc40-4513-840f-c8bfc3761c51 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:03.718904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd00606c-dc40-4513-840f-c8bfc3761c51 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:03.720528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dd00606c-dc40-4513-840f-c8bfc3761c51 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:03.822434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd00606c-dc40-4513-840f-c8bfc3761c51 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:03.824060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd00606c-dc40-4513-840f-c8bfc3761c51 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:03.842284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 716/1419] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:04.992866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fe768-a231-4b33-ae94-545f51d7e110 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:04.993407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fe768-a231-4b33-ae94-545f51d7e110 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:04.993407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fe768-a231-4b33-ae94-545f51d7e110 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:04.994698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 704/1420] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:05.002191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa45d4-ed0b-46d2-8477-7f279b83641e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:05.002665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa45d4-ed0b-46d2-8477-7f279b83641e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:05.002785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa45d4-ed0b-46d2-8477-7f279b83641e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:05.073174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:05.096220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7cfa45d4-ed0b-46d2-8477-7f279b83641e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:05.097462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7cfa45d4-ed0b-46d2-8477-7f279b83641e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:05.194470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 717/1421] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:05.344139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861fe768-a231-4b33-ae94-545f51d7e110 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:05.345650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-861fe768-a231-4b33-ae94-545f51d7e110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:05.506531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861fe768-a231-4b33-ae94-545f51d7e110 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:05.519195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:05.622179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7038c29e-66d3-41a4-b6f9-a2d6c0ff0952 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:05.622712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7038c29e-66d3-41a4-b6f9-a2d6c0ff0952 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:05.623009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7038c29e-66d3-41a4-b6f9-a2d6c0ff0952 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:05.623281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7038c29e-66d3-41a4-b6f9-a2d6c0ff0952 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:05.624854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7038c29e-66d3-41a4-b6f9-a2d6c0ff0952 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:05.712655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7038c29e-66d3-41a4-b6f9-a2d6c0ff0952 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:05.714271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7038c29e-66d3-41a4-b6f9-a2d6c0ff0952 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:05.732857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 705/1422] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:06.898824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7865502-7752-4334-bf6a-61469c53de63 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:06.899418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7865502-7752-4334-bf6a-61469c53de63 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:06.899418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7865502-7752-4334-bf6a-61469c53de63 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:06.900766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 718/1423] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:06.907322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f27dd4-1f82-48e9-b03c-b4c8054c1a4c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:06.907768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f27dd4-1f82-48e9-b03c-b4c8054c1a4c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:06.907877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f27dd4-1f82-48e9-b03c-b4c8054c1a4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:06.928166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:06.985435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-32f27dd4-1f82-48e9-b03c-b4c8054c1a4c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:06.986874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-32f27dd4-1f82-48e9-b03c-b4c8054c1a4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:07.086506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 706/1424] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:07.241920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7865502-7752-4334-bf6a-61469c53de63 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:07.243421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f7865502-7752-4334-bf6a-61469c53de63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:07.352409 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:07.390799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7865502-7752-4334-bf6a-61469c53de63 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:07.502998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4d6db9-22b0-4857-b99b-18be8030d569 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:07.503516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4d6db9-22b0-4857-b99b-18be8030d569 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:07.503787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4d6db9-22b0-4857-b99b-18be8030d569 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:07.504051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4d6db9-22b0-4857-b99b-18be8030d569 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:07.505629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fb4d6db9-22b0-4857-b99b-18be8030d569 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:07.581899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:07.596051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4d6db9-22b0-4857-b99b-18be8030d569 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:07.597650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4d6db9-22b0-4857-b99b-18be8030d569 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:07.615815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 719/1425] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:08.718146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe22245-6f2c-4bab-91d6-ec1d212ae30f None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:08.718623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe22245-6f2c-4bab-91d6-ec1d212ae30f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:08.718722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe22245-6f2c-4bab-91d6-ec1d212ae30f None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:08.720067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 707/1426] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:08.726330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9716aa-1a6c-4b46-8d93-b0943ff7f383 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:08.726778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9716aa-1a6c-4b46-8d93-b0943ff7f383 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:08.726890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9716aa-1a6c-4b46-8d93-b0943ff7f383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:08.801075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-5d9716aa-1a6c-4b46-8d93-b0943ff7f383 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:08.802267 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5d9716aa-1a6c-4b46-8d93-b0943ff7f383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:08.896850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 720/1427] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:09.049761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe22245-6f2c-4bab-91d6-ec1d212ae30f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:09.051304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8fe22245-6f2c-4bab-91d6-ec1d212ae30f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:09.136698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:09.200073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe22245-6f2c-4bab-91d6-ec1d212ae30f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:09.314106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1c1514-62c4-4aa8-85e0-89805171ba88 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:09.314643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1c1514-62c4-4aa8-85e0-89805171ba88 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:09.314916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1c1514-62c4-4aa8-85e0-89805171ba88 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:09.315208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1c1514-62c4-4aa8-85e0-89805171ba88 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:09.316791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6c1c1514-62c4-4aa8-85e0-89805171ba88 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:09.364194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:09.407786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1c1514-62c4-4aa8-85e0-89805171ba88 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:09.409410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1c1514-62c4-4aa8-85e0-89805171ba88 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:09.428158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 708/1428] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:10.474506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93745aa-720b-4c55-91c7-71204ee6eb07 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:10.474947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93745aa-720b-4c55-91c7-71204ee6eb07 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:10.475082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93745aa-720b-4c55-91c7-71204ee6eb07 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:10.476337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 721/1429] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:10.483558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952dc3b7-e471-407f-a9b5-f711defa8d48 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:10.484012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952dc3b7-e471-407f-a9b5-f711defa8d48 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:10.484109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952dc3b7-e471-407f-a9b5-f711defa8d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:10.558845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-952dc3b7-e471-407f-a9b5-f711defa8d48 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:10.560265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-952dc3b7-e471-407f-a9b5-f711defa8d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:10.654532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 709/1430] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:10.803524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93745aa-720b-4c55-91c7-71204ee6eb07 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:10.804964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e93745aa-720b-4c55-91c7-71204ee6eb07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:10.853259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:10.951189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93745aa-720b-4c55-91c7-71204ee6eb07 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:11.062119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a34cfa-47ee-4917-b6a5-26911799f32a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:11.062627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a34cfa-47ee-4917-b6a5-26911799f32a service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:11.062905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a34cfa-47ee-4917-b6a5-26911799f32a service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:11.063205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a34cfa-47ee-4917-b6a5-26911799f32a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:11.064795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a0a34cfa-47ee-4917-b6a5-26911799f32a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:11.071603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:11.155505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a34cfa-47ee-4917-b6a5-26911799f32a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:11.157111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a34cfa-47ee-4917-b6a5-26911799f32a service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:11.174842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 722/1431] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:12.227091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38208df-4cc9-48d6-90c9-53f170d2a23f None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:12.227588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38208df-4cc9-48d6-90c9-53f170d2a23f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:12.227682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38208df-4cc9-48d6-90c9-53f170d2a23f None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:12.229051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 710/1432] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:12.235441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4586426-2c3a-4d69-9560-fc27d8ceb88e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:12.235906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4586426-2c3a-4d69-9560-fc27d8ceb88e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:12.236003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4586426-2c3a-4d69-9560-fc27d8ceb88e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:12.311841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b4586426-2c3a-4d69-9560-fc27d8ceb88e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:12.313029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b4586426-2c3a-4d69-9560-fc27d8ceb88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:12.380681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:12.410392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 723/1433] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:12.559340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38208df-4cc9-48d6-90c9-53f170d2a23f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:12.561380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d38208df-4cc9-48d6-90c9-53f170d2a23f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:12.578765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:12.706155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38208df-4cc9-48d6-90c9-53f170d2a23f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:12.792300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:12.821304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef9531-b0ca-4f0a-a8d2-a6a145a312f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:12.821908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef9531-b0ca-4f0a-a8d2-a6a145a312f3 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:12.822178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef9531-b0ca-4f0a-a8d2-a6a145a312f3 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:12.822443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef9531-b0ca-4f0a-a8d2-a6a145a312f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:12.824063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-41ef9531-b0ca-4f0a-a8d2-a6a145a312f3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:12.915064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ef9531-b0ca-4f0a-a8d2-a6a145a312f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:12.916713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ef9531-b0ca-4f0a-a8d2-a6a145a312f3 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:12.934466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 711/1434] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:13.983915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660e8ccf-11da-4f39-84a3-d860a96b99c6 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:13.984508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660e8ccf-11da-4f39-84a3-d860a96b99c6 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:13.984508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660e8ccf-11da-4f39-84a3-d860a96b99c6 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:13.985778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 724/1435] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:13.992144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a177b8b-686d-44f8-9cc9-21262e6f69b0 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:13.992620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a177b8b-686d-44f8-9cc9-21262e6f69b0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:13.992733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a177b8b-686d-44f8-9cc9-21262e6f69b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:14.070370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1a177b8b-686d-44f8-9cc9-21262e6f69b0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:14.071597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1a177b8b-686d-44f8-9cc9-21262e6f69b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:14.113751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:14.170110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 712/1436] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:14.320600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660e8ccf-11da-4f39-84a3-d860a96b99c6 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:14.322113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-660e8ccf-11da-4f39-84a3-d860a96b99c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:14.466228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660e8ccf-11da-4f39-84a3-d860a96b99c6 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:14.519591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:14.577635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b491e2c-e014-4eae-8aef-8378589c17a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:14.578166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b491e2c-e014-4eae-8aef-8378589c17a7 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:14.578439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b491e2c-e014-4eae-8aef-8378589c17a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:14.578707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b491e2c-e014-4eae-8aef-8378589c17a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:14.580337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0b491e2c-e014-4eae-8aef-8378589c17a7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:14.669679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b491e2c-e014-4eae-8aef-8378589c17a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:14.671296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b491e2c-e014-4eae-8aef-8378589c17a7 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:14.689289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 725/1437] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:26:15.744786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39876cbe-6047-43a6-aff9-c942d5a4a821 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:15.745378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39876cbe-6047-43a6-aff9-c942d5a4a821 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:15.745378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39876cbe-6047-43a6-aff9-c942d5a4a821 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:15.746697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 713/1438] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:26:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:15.753600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c439534-1dd0-4a71-ae51-ec51dd2dd7b2 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:15.754045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c439534-1dd0-4a71-ae51-ec51dd2dd7b2 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:15.754152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c439534-1dd0-4a71-ae51-ec51dd2dd7b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:15.829159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9c439534-1dd0-4a71-ae51-ec51dd2dd7b2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:15.830374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9c439534-1dd0-4a71-ae51-ec51dd2dd7b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:15.832394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:15.930109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 726/1439] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:16.084271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39876cbe-6047-43a6-aff9-c942d5a4a821 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:16.085798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-39876cbe-6047-43a6-aff9-c942d5a4a821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:16.231943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39876cbe-6047-43a6-aff9-c942d5a4a821 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:16.260789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:16.349624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b4c66e-3fb9-4abf-ac81-d81697aebf94 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:16.350123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b4c66e-3fb9-4abf-ac81-d81697aebf94 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:16.350397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b4c66e-3fb9-4abf-ac81-d81697aebf94 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:16.350667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b4c66e-3fb9-4abf-ac81-d81697aebf94 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:16.352312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b6b4c66e-3fb9-4abf-ac81-d81697aebf94 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:16.440672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b4c66e-3fb9-4abf-ac81-d81697aebf94 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:16.442280 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b4c66e-3fb9-4abf-ac81-d81697aebf94 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:16.460363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 714/1440] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:18.534212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df22498-be1a-456b-9bc2-b18fc6b87702 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:18.534780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df22498-be1a-456b-9bc2-b18fc6b87702 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:18.534780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df22498-be1a-456b-9bc2-b18fc6b87702 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:18.536034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 727/1441] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:18.541216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1523b6-1243-4752-a313-243ec79a00f3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:18.541661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1523b6-1243-4752-a313-243ec79a00f3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:18.541766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1523b6-1243-4752-a313-243ec79a00f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:18.582926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:18.608647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-db1523b6-1243-4752-a313-243ec79a00f3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:18.609848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-db1523b6-1243-4752-a313-243ec79a00f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:18.706505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 715/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:18.991425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df22498-be1a-456b-9bc2-b18fc6b87702 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:18.992961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7df22498-be1a-456b-9bc2-b18fc6b87702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:19.095117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df22498-be1a-456b-9bc2-b18fc6b87702 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:19.131967 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:19.174488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf49ce6-f703-4eeb-bc7a-ca743c09d7ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:19.175130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf49ce6-f703-4eeb-bc7a-ca743c09d7ea service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:19.175435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf49ce6-f703-4eeb-bc7a-ca743c09d7ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:19.175707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf49ce6-f703-4eeb-bc7a-ca743c09d7ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:19.177312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9bf49ce6-f703-4eeb-bc7a-ca743c09d7ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:19.288751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf49ce6-f703-4eeb-bc7a-ca743c09d7ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:19.290096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf49ce6-f703-4eeb-bc7a-ca743c09d7ea service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:19.307609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 728/1443] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:21.623724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1908e1d-112f-4601-b50e-d9c49cec5ec5 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:21.624265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1908e1d-112f-4601-b50e-d9c49cec5ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:21.624265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1908e1d-112f-4601-b50e-d9c49cec5ec5 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:21.625661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 716/1444] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:21.631046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb6777-cbdf-47a1-b775-551fd644abec None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:21.631508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb6777-cbdf-47a1-b775-551fd644abec None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:21.631632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cb6777-cbdf-47a1-b775-551fd644abec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:21.698394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e1cb6777-cbdf-47a1-b775-551fd644abec None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:21.699943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e1cb6777-cbdf-47a1-b775-551fd644abec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:21.715543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:21.803037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 729/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:22.103044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1908e1d-112f-4601-b50e-d9c49cec5ec5 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:22.104623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c1908e1d-112f-4601-b50e-d9c49cec5ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:22.208269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1908e1d-112f-4601-b50e-d9c49cec5ec5 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:22.221453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:22.283607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fed4ba9-1a21-4622-8279-28d42943cf12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:22.284209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fed4ba9-1a21-4622-8279-28d42943cf12 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:22.284483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fed4ba9-1a21-4622-8279-28d42943cf12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:22.284752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fed4ba9-1a21-4622-8279-28d42943cf12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:22.286344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2fed4ba9-1a21-4622-8279-28d42943cf12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:22.397679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fed4ba9-1a21-4622-8279-28d42943cf12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:22.399064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fed4ba9-1a21-4622-8279-28d42943cf12 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:22.419630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 717/1446] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:24.900611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b517eae-0e1b-46e4-aee7-904fdbf65129 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:24.901156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b517eae-0e1b-46e4-aee7-904fdbf65129 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:24.901156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b517eae-0e1b-46e4-aee7-904fdbf65129 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:24.902460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 730/1447] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:24.908927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca99aad-6a58-4e41-b165-a3bab07c4bea None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:24.909400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca99aad-6a58-4e41-b165-a3bab07c4bea None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:24.909517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca99aad-6a58-4e41-b165-a3bab07c4bea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:24.958779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:24.977893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1ca99aad-6a58-4e41-b165-a3bab07c4bea None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:24.979158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1ca99aad-6a58-4e41-b165-a3bab07c4bea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:25.076111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 718/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:25.366297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b517eae-0e1b-46e4-aee7-904fdbf65129 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:25.367878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7b517eae-0e1b-46e4-aee7-904fdbf65129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:25.470250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b517eae-0e1b-46e4-aee7-904fdbf65129 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:25.515396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:25.552364 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6688ea-e136-4a66-870d-62c6a6d125eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:25.553060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6688ea-e136-4a66-870d-62c6a6d125eb service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:25.553417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6688ea-e136-4a66-870d-62c6a6d125eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:25.553747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6688ea-e136-4a66-870d-62c6a6d125eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:25.555454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dd6688ea-e136-4a66-870d-62c6a6d125eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:25.671529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6688ea-e136-4a66-870d-62c6a6d125eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:25.672958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6688ea-e136-4a66-870d-62c6a6d125eb service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:25.691901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 731/1449] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:27.993873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c4b30a-c6fe-41f3-82f6-e6cfc21d8a6c None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:27.994445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c4b30a-c6fe-41f3-82f6-e6cfc21d8a6c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:27.994445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c4b30a-c6fe-41f3-82f6-e6cfc21d8a6c None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:27.995794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 719/1450] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:28.001649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a33373-39f6-41e1-994a-500f74456514 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:28.002068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a33373-39f6-41e1-994a-500f74456514 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:28.002197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a33373-39f6-41e1-994a-500f74456514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:28.067413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-90a33373-39f6-41e1-994a-500f74456514 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:28.068633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-90a33373-39f6-41e1-994a-500f74456514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:28.089438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:28.165706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 732/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:28.452374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c4b30a-c6fe-41f3-82f6-e6cfc21d8a6c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:28.453920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e7c4b30a-c6fe-41f3-82f6-e6cfc21d8a6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:28.558229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c4b30a-c6fe-41f3-82f6-e6cfc21d8a6c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:28.579174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:28.634243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db46cea1-5925-4a66-8fd7-014b6e39502d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:28.634841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46cea1-5925-4a66-8fd7-014b6e39502d service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:28.635155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46cea1-5925-4a66-8fd7-014b6e39502d service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:28.635429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46cea1-5925-4a66-8fd7-014b6e39502d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:28.637026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-db46cea1-5925-4a66-8fd7-014b6e39502d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:28.749492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db46cea1-5925-4a66-8fd7-014b6e39502d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:28.750932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db46cea1-5925-4a66-8fd7-014b6e39502d service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:28.768866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 720/1452] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:30.973173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72954b94-f7a4-4ad5-afe9-dcb093060347 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:30.973699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72954b94-f7a4-4ad5-afe9-dcb093060347 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:30.973821 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72954b94-f7a4-4ad5-afe9-dcb093060347 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:30.975119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 733/1453] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:30.980555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9afa05e-7709-4e4c-b34e-56553bb982d9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:30.981093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9afa05e-7709-4e4c-b34e-56553bb982d9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:30.981093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9afa05e-7709-4e4c-b34e-56553bb982d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:31.044776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:31.048206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b9afa05e-7709-4e4c-b34e-56553bb982d9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:31.050065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b9afa05e-7709-4e4c-b34e-56553bb982d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:31.146076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 721/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:31.430111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72954b94-f7a4-4ad5-afe9-dcb093060347 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:31.431673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-72954b94-f7a4-4ad5-afe9-dcb093060347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:31.533731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72954b94-f7a4-4ad5-afe9-dcb093060347 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:31.581964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:31.606316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadb4b2e-d378-48af-b88a-ce1eb567bab0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:31.606906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb4b2e-d378-48af-b88a-ce1eb567bab0 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:31.607206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb4b2e-d378-48af-b88a-ce1eb567bab0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:31.607471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb4b2e-d378-48af-b88a-ce1eb567bab0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:31.609285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-cadb4b2e-d378-48af-b88a-ce1eb567bab0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:31.717446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadb4b2e-d378-48af-b88a-ce1eb567bab0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:31.718869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadb4b2e-d378-48af-b88a-ce1eb567bab0 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:31.736735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 734/1455] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:33.942021 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372d020-80b0-47b7-a029-9ecf793cd9ad None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:33.942566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372d020-80b0-47b7-a029-9ecf793cd9ad None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:33.942698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372d020-80b0-47b7-a029-9ecf793cd9ad None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:33.944001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 722/1456] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:33.949341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c02011-016a-4f0d-a0cf-981a1c95c50b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:33.949740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c02011-016a-4f0d-a0cf-981a1c95c50b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:33.949884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c02011-016a-4f0d-a0cf-981a1c95c50b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:34.016354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d2c02011-016a-4f0d-a0cf-981a1c95c50b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:34.017553 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d2c02011-016a-4f0d-a0cf-981a1c95c50b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:34.046548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:34.114803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 735/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:34.403186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8372d020-80b0-47b7-a029-9ecf793cd9ad None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:34.404830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8372d020-80b0-47b7-a029-9ecf793cd9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:34.508483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8372d020-80b0-47b7-a029-9ecf793cd9ad None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:34.536716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:34.582290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdccc76-de85-4eb5-9682-9e32eb2a9433 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:34.582890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdccc76-de85-4eb5-9682-9e32eb2a9433 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:34.583192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdccc76-de85-4eb5-9682-9e32eb2a9433 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:34.583463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdccc76-de85-4eb5-9682-9e32eb2a9433 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:34.585035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fcdccc76-de85-4eb5-9682-9e32eb2a9433 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:34.694095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdccc76-de85-4eb5-9682-9e32eb2a9433 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:34.695641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcdccc76-de85-4eb5-9682-9e32eb2a9433 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:34.718566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 723/1458] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:36.919131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ef409-d1d1-494e-a9f9-112b39e95f86 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:36.919757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ef409-d1d1-494e-a9f9-112b39e95f86 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:36.919757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ef409-d1d1-494e-a9f9-112b39e95f86 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:36.921842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 736/1459] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:36.927138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116412ed-4b7a-460d-a65d-fae092114c05 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:36.927555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116412ed-4b7a-460d-a65d-fae092114c05 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:36.927657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116412ed-4b7a-460d-a65d-fae092114c05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:36.995964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-116412ed-4b7a-460d-a65d-fae092114c05 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:36.997100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-116412ed-4b7a-460d-a65d-fae092114c05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:37.005529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:37.099096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 724/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:37.387697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39ef409-d1d1-494e-a9f9-112b39e95f86 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:37.389768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c39ef409-d1d1-494e-a9f9-112b39e95f86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:37.495738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39ef409-d1d1-494e-a9f9-112b39e95f86 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:37.554004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:37.572479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b65a8a-cec6-4f20-8a26-fc24318b5626 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:37.573107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b65a8a-cec6-4f20-8a26-fc24318b5626 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:37.573384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b65a8a-cec6-4f20-8a26-fc24318b5626 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:37.573660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b65a8a-cec6-4f20-8a26-fc24318b5626 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:37.575279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-28b65a8a-cec6-4f20-8a26-fc24318b5626 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:37.691722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b65a8a-cec6-4f20-8a26-fc24318b5626 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:37.693207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b65a8a-cec6-4f20-8a26-fc24318b5626 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:37.711063 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 737/1461] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:39.897389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7f6d5-f358-4be9-8841-7a6bb41b02d2 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:39.897985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7f6d5-f358-4be9-8841-7a6bb41b02d2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:39.897985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be7f6d5-f358-4be9-8841-7a6bb41b02d2 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:39.899354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 725/1462] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:39.904447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6d048e-beb4-42f4-b3e8-422424d24989 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:39.905408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6d048e-beb4-42f4-b3e8-422424d24989 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:39.905516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6d048e-beb4-42f4-b3e8-422424d24989 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:39.972767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-de6d048e-beb4-42f4-b3e8-422424d24989 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:39.973983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-de6d048e-beb4-42f4-b3e8-422424d24989 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:40.012508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:40.070376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 738/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:40.354355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be7f6d5-f358-4be9-8841-7a6bb41b02d2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:40.355961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5be7f6d5-f358-4be9-8841-7a6bb41b02d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:40.463589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be7f6d5-f358-4be9-8841-7a6bb41b02d2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:40.502814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:40.538673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37610740-cdd4-498d-9571-61ec61bffe30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:40.539306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37610740-cdd4-498d-9571-61ec61bffe30 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:40.539580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37610740-cdd4-498d-9571-61ec61bffe30 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:40.539843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37610740-cdd4-498d-9571-61ec61bffe30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:40.541431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-37610740-cdd4-498d-9571-61ec61bffe30 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:40.649329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37610740-cdd4-498d-9571-61ec61bffe30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:40.650762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37610740-cdd4-498d-9571-61ec61bffe30 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:40.668282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 726/1464] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:42.854461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df116a-8a6c-40ac-b10c-928d25c71ecc None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:42.855018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df116a-8a6c-40ac-b10c-928d25c71ecc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:42.855018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7df116a-8a6c-40ac-b10c-928d25c71ecc None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:42.856373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 739/1465] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:42.862281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411ba883-eb20-4210-933f-7242021a397b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:42.862935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411ba883-eb20-4210-933f-7242021a397b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:42.863122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411ba883-eb20-4210-933f-7242021a397b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:42.929845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-411ba883-eb20-4210-933f-7242021a397b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:42.931089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-411ba883-eb20-4210-933f-7242021a397b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:42.948685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:43.027808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 727/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:43.311340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df116a-8a6c-40ac-b10c-928d25c71ecc None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:43.312925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d7df116a-8a6c-40ac-b10c-928d25c71ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:43.414797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7df116a-8a6c-40ac-b10c-928d25c71ecc None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:43.481337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:43.489477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6f568c-d06f-42aa-a2ad-89faf5e1365d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:43.490066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f568c-d06f-42aa-a2ad-89faf5e1365d service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:43.490339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f568c-d06f-42aa-a2ad-89faf5e1365d service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:43.490602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f568c-d06f-42aa-a2ad-89faf5e1365d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:43.492231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ef6f568c-d06f-42aa-a2ad-89faf5e1365d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:43.602252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6f568c-d06f-42aa-a2ad-89faf5e1365d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:43.603707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6f568c-d06f-42aa-a2ad-89faf5e1365d service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:43.622199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 740/1467] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:45.793424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee8ac0-05ed-4e85-af28-48e5c1df3c94 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:45.794034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee8ac0-05ed-4e85-af28-48e5c1df3c94 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:45.794138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee8ac0-05ed-4e85-af28-48e5c1df3c94 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:45.796766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 728/1468] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:26:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:26:45.802022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28841e98-e52f-438f-bb01-0b6f2b9a2b30 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:45.802472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28841e98-e52f-438f-bb01-0b6f2b9a2b30 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:45.802577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28841e98-e52f-438f-bb01-0b6f2b9a2b30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:45.868733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-28841e98-e52f-438f-bb01-0b6f2b9a2b30 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:26:45.869926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-28841e98-e52f-438f-bb01-0b6f2b9a2b30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:45.918207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:45.966094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 741/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:26:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:26:46.250917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee8ac0-05ed-4e85-af28-48e5c1df3c94 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:26:46.252486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-09ee8ac0-05ed-4e85-af28-48e5c1df3c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:46.355249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee8ac0-05ed-4e85-af28-48e5c1df3c94 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:26:46.407345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:46.433269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65733c86-a867-40cd-b65f-86f2a5d78e9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:26:46.433867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65733c86-a867-40cd-b65f-86f2a5d78e9a service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:26:46.434138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65733c86-a867-40cd-b65f-86f2a5d78e9a service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:26:46.434400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65733c86-a867-40cd-b65f-86f2a5d78e9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:26:46.436035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-65733c86-a867-40cd-b65f-86f2a5d78e9a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:26:46.544397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65733c86-a867-40cd-b65f-86f2a5d78e9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:26:46.546418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65733c86-a867-40cd-b65f-86f2a5d78e9a service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:26:46.564018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 729/1470] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:26:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:26:55.964599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=22 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:26:56.561790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=66 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:06.683289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855305d5-6a5f-4e1f-b4bf-263a4cf0f8ca None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:06.683837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855305d5-6a5f-4e1f-b4bf-263a4cf0f8ca None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:06.683837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855305d5-6a5f-4e1f-b4bf-263a4cf0f8ca None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:06.685164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 742/1471] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:27:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:06.691140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac2a5a4-a813-470a-b459-ee004ea01f73 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:06.691595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac2a5a4-a813-470a-b459-ee004ea01f73 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:06.691694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac2a5a4-a813-470a-b459-ee004ea01f73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:06.766163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-bac2a5a4-a813-470a-b459-ee004ea01f73 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:06.767409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-bac2a5a4-a813-470a-b459-ee004ea01f73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:06.863628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 730/1472] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:27:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:07.171953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696bbe9c-0f98-4dbd-8fc0-bb64acc6f047 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:07.172240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696bbe9c-0f98-4dbd-8fc0-bb64acc6f047 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:07.172519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696bbe9c-0f98-4dbd-8fc0-bb64acc6f047 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:07.173845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 743/1473] 10.222.0.33 () {54 vars in 962 bytes} [Thu Oct 6 04:27:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:07.178827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa33b2-5350-4a89-ac3f-47d70ccdd17d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:07.179142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa33b2-5350-4a89-ac3f-47d70ccdd17d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:07.179414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa33b2-5350-4a89-ac3f-47d70ccdd17d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:07.255845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-13aa33b2-5350-4a89-ac3f-47d70ccdd17d None None] MFA Rules not processed for user `b34c61ebdb9147d5b3d66cbfa556f38e`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:07.257026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-13aa33b2-5350-4a89-ac3f-47d70ccdd17d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:07.277478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:07.333250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 731/1474] 10.222.0.33 () {58 vars in 1059 bytes} [Thu Oct 6 04:27:07 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:07.340863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696bbe9c-0f98-4dbd-8fc0-bb64acc6f047 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:27:07.342953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-696bbe9c-0f98-4dbd-8fc0-bb64acc6f047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:07.456013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696bbe9c-0f98-4dbd-8fc0-bb64acc6f047 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:27:07.531390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6925f0a7-2e4e-4c4e-b174-32870ad33175 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b34c61ebdb9147d5b3d66cbfa556f38e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:27:07.531989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6925f0a7-2e4e-4c4e-b174-32870ad33175 service glance] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:07.532263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6925f0a7-2e4e-4c4e-b174-32870ad33175 service glance] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:07.532532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6925f0a7-2e4e-4c4e-b174-32870ad33175 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:07.534114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6925f0a7-2e4e-4c4e-b174-32870ad33175 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:07.593794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:07.648106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6925f0a7-2e4e-4c4e-b174-32870ad33175 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:27:07.649561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6925f0a7-2e4e-4c4e-b174-32870ad33175 service glance] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:27:07.668099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 744/1475] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:27:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:27:10.012806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb05a1-a308-418d-bdf6-3a6a05428aa8 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:10.013424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb05a1-a308-418d-bdf6-3a6a05428aa8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:10.013424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eb05a1-a308-418d-bdf6-3a6a05428aa8 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:10.015300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 732/1476] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:27:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:10.020720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df6c86-c9c0-48d9-99e1-c784490d5e23 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:10.021187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df6c86-c9c0-48d9-99e1-c784490d5e23 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:10.021382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75df6c86-c9c0-48d9-99e1-c784490d5e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:10.204769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-75df6c86-c9c0-48d9-99e1-c784490d5e23 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:10.205993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-75df6c86-c9c0-48d9-99e1-c784490d5e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:10.300518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 745/1477] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:27:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:10.580333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6eb05a1-a308-418d-bdf6-3a6a05428aa8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:27:10.581867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c6eb05a1-a308-418d-bdf6-3a6a05428aa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:10.705273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6eb05a1-a308-418d-bdf6-3a6a05428aa8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:27:10.756118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:10.799220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3557d8-e878-414f-8d62-f3ee8abc3c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:27:10.799819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3557d8-e878-414f-8d62-f3ee8abc3c69 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:10.800093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3557d8-e878-414f-8d62-f3ee8abc3c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:10.800362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3557d8-e878-414f-8d62-f3ee8abc3c69 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:10.800997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3557d8-e878-414f-8d62-f3ee8abc3c69 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:27:10.802771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-4e3557d8-e878-414f-8d62-f3ee8abc3c69 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:27:10.802771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:27:10.802771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:27:10.803963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3557d8-e878-414f-8d62-f3ee8abc3c69 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:27:10.823139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 733/1478] 10.222.0.33 () {56 vars in 1108 bytes} [Thu Oct 6 04:27:10 2022] GET /identity/v3/projects => generated 2553 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:27:13.035926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89fa3a-a16f-49c3-99d2-8e7db4de2a63 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:13.036454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89fa3a-a16f-49c3-99d2-8e7db4de2a63 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:13.036454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89fa3a-a16f-49c3-99d2-8e7db4de2a63 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:13.037755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 746/1479] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:27:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:13.043050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176f8e17-99f3-4a93-a5bc-d2c4f40d0684 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:13.043449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176f8e17-99f3-4a93-a5bc-d2c4f40d0684 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:13.043624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176f8e17-99f3-4a93-a5bc-d2c4f40d0684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:13.109578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-176f8e17-99f3-4a93-a5bc-d2c4f40d0684 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:13.110777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-176f8e17-99f3-4a93-a5bc-d2c4f40d0684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:13.206478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 734/1480] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:27:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:13.504938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89fa3a-a16f-49c3-99d2-8e7db4de2a63 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:27:13.506700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7c89fa3a-a16f-49c3-99d2-8e7db4de2a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:13.518265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:13.652377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89fa3a-a16f-49c3-99d2-8e7db4de2a63 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:27:13.731288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:13.764226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f240b858-ec24-45f5-a72c-9d70095fb3ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:27:13.764747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f240b858-ec24-45f5-a72c-9d70095fb3ec service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:13.765016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f240b858-ec24-45f5-a72c-9d70095fb3ec service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:13.765280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f240b858-ec24-45f5-a72c-9d70095fb3ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:13.766850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f240b858-ec24-45f5-a72c-9d70095fb3ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:13.854702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f240b858-ec24-45f5-a72c-9d70095fb3ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:27:13.856302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f240b858-ec24-45f5-a72c-9d70095fb3ec service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:27:13.874777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 747/1481] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:27:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:27:16.212310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f755dcee-4d15-47a6-aebc-a14e988df642 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:16.212883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f755dcee-4d15-47a6-aebc-a14e988df642 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:16.213133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f755dcee-4d15-47a6-aebc-a14e988df642 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:16.214803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 735/1482] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:27:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:16.221013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d81a3b-71d3-4fb8-817d-cda539b001b7 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:16.221444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d81a3b-71d3-4fb8-817d-cda539b001b7 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:16.221540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d81a3b-71d3-4fb8-817d-cda539b001b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:16.287945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f1d81a3b-71d3-4fb8-817d-cda539b001b7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:16.289140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f1d81a3b-71d3-4fb8-817d-cda539b001b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:16.334183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:16.384096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 748/1483] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:27:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:16.673345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f755dcee-4d15-47a6-aebc-a14e988df642 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:27:16.674884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f755dcee-4d15-47a6-aebc-a14e988df642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:16.699477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:16.784607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f755dcee-4d15-47a6-aebc-a14e988df642 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:27:16.858042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d5467f-979f-445e-bfae-b6870439dd31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:27:16.858628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5467f-979f-445e-bfae-b6870439dd31 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:16.858900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5467f-979f-445e-bfae-b6870439dd31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:16.859193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5467f-979f-445e-bfae-b6870439dd31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:16.860758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c3d5467f-979f-445e-bfae-b6870439dd31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:16.936829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:16.971695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d5467f-979f-445e-bfae-b6870439dd31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:27:16.973143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d5467f-979f-445e-bfae-b6870439dd31 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:27:16.992161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 736/1484] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:27:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:27:19.175037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ab441-acbf-40cd-b0de-b9f5998aafec None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:19.175513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ab441-acbf-40cd-b0de-b9f5998aafec None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:19.175624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ab441-acbf-40cd-b0de-b9f5998aafec None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:19.176926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 749/1485] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:27:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:19.182028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5746b-bf2b-4d7c-b876-52da30b26adc None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:19.182468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5746b-bf2b-4d7c-b876-52da30b26adc None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:19.182559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5746b-bf2b-4d7c-b876-52da30b26adc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:19.249137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-74c5746b-bf2b-4d7c-b876-52da30b26adc None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:19.250318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-74c5746b-bf2b-4d7c-b876-52da30b26adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:19.344265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 737/1486] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:19.632727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03ab441-acbf-40cd-b0de-b9f5998aafec None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:27:19.634301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e03ab441-acbf-40cd-b0de-b9f5998aafec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:19.737651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03ab441-acbf-40cd-b0de-b9f5998aafec None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:27:19.808234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:19.809695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c50d01-8cec-43b9-abfa-3155bf1da6dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:27:19.810798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c50d01-8cec-43b9-abfa-3155bf1da6dd service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:19.811120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c50d01-8cec-43b9-abfa-3155bf1da6dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:19.811639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c50d01-8cec-43b9-abfa-3155bf1da6dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:19.813602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f5c50d01-8cec-43b9-abfa-3155bf1da6dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:19.920577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c50d01-8cec-43b9-abfa-3155bf1da6dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:27:19.922074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c50d01-8cec-43b9-abfa-3155bf1da6dd service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:27:19.942033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 750/1487] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:27:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:27:22.226085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f83cc-ed4e-4477-a25a-f27b068c8629 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:22.226583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f83cc-ed4e-4477-a25a-f27b068c8629 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:22.226682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f83cc-ed4e-4477-a25a-f27b068c8629 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:22.228010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 738/1488] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:27:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:22.233362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9b3d0a-07da-4cea-8eed-246283584c4c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:22.233817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9b3d0a-07da-4cea-8eed-246283584c4c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:22.233980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9b3d0a-07da-4cea-8eed-246283584c4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:22.299455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6e9b3d0a-07da-4cea-8eed-246283584c4c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:22.300651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6e9b3d0a-07da-4cea-8eed-246283584c4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:22.356937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:22.402145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 751/1489] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:27:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:22.709605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3f83cc-ed4e-4477-a25a-f27b068c8629 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:27:22.711215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7c3f83cc-ed4e-4477-a25a-f27b068c8629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:22.749753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:22.820135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3f83cc-ed4e-4477-a25a-f27b068c8629 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:27:22.894387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d4349b-e712-423f-b7d2-67c6b5ce7bf9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:27:22.895016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4349b-e712-423f-b7d2-67c6b5ce7bf9 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:22.895298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4349b-e712-423f-b7d2-67c6b5ce7bf9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:22.895568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4349b-e712-423f-b7d2-67c6b5ce7bf9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:22.897170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-51d4349b-e712-423f-b7d2-67c6b5ce7bf9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:22.980558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:23.012221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d4349b-e712-423f-b7d2-67c6b5ce7bf9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:27:23.013805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d4349b-e712-423f-b7d2-67c6b5ce7bf9 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:27:23.153354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 739/1490] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:27:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:27:25.615380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a2581c-4b5a-4c7f-9fda-2cc1a056601d None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:25.615911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a2581c-4b5a-4c7f-9fda-2cc1a056601d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:25.615911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a2581c-4b5a-4c7f-9fda-2cc1a056601d None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:25.617265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 752/1491] 10.222.0.33 () {54 vars in 927 bytes} [Thu Oct 6 04:27:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:27:25.622475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7215d-bc6b-4a8e-8c50-103f1e669b57 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:25.623604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7215d-bc6b-4a8e-8c50-103f1e669b57 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:25.623604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7215d-bc6b-4a8e-8c50-103f1e669b57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:25.689823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e7f7215d-bc6b-4a8e-8c50-103f1e669b57 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:27:25.691055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e7f7215d-bc6b-4a8e-8c50-103f1e669b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:25.790389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 740/1492] 10.222.0.33 () {58 vars in 1024 bytes} [Thu Oct 6 04:27:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:27:26.082092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a2581c-4b5a-4c7f-9fda-2cc1a056601d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:27:26.083772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-00a2581c-4b5a-4c7f-9fda-2cc1a056601d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:26.185890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a2581c-4b5a-4c7f-9fda-2cc1a056601d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:27:26.259416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f53ae2d-1a42-4db0-b307-3ecf6c864664 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:27:26.260012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f53ae2d-1a42-4db0-b307-3ecf6c864664 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:27:26.260285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f53ae2d-1a42-4db0-b307-3ecf6c864664 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:27:26.260553 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f53ae2d-1a42-4db0-b307-3ecf6c864664 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:27:26.262144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0f53ae2d-1a42-4db0-b307-3ecf6c864664 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:27:26.270551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:26.369640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f53ae2d-1a42-4db0-b307-3ecf6c864664 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:27:26.370982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f53ae2d-1a42-4db0-b307-3ecf6c864664 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:27:26.389492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 753/1493] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:27:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:27:35.788497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=82 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:27:36.386921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=50 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:46.078896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:46.079533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:46.079533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:46.156437 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:28:46.157624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:46.252252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 741/1494] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:28:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:28:46.259984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98248e2e-7310-4124-aa51-0f64fff21c17 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:46.260458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98248e2e-7310-4124-aa51-0f64fff21c17 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:46.260559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98248e2e-7310-4124-aa51-0f64fff21c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:46.336371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-98248e2e-7310-4124-aa51-0f64fff21c17 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:28:46.337490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-98248e2e-7310-4124-aa51-0f64fff21c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:46.434852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 754/1495] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:28:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:28:46.439520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:46.440717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:46.451922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c6fb6e-fb9b-4284-a08b-c14f1b2efa7f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:46.513765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:46.547744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:46.548343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:46.548618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:46.548902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:46.549515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:28:46.549809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:46.550810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:46.550810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:28:46.550810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:28:46.551729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:46.563405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 742/1496] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:28:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:28:46.571683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98248e2e-7310-4124-aa51-0f64fff21c17 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:46.580958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98248e2e-7310-4124-aa51-0f64fff21c17 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:46.640845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:46.673732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:46.674317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:46.674592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:46.674856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:46.675507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:46.677223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING py.warnings [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:46.677223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: warnings.warn(msg) Oct 06 04:28:46.677223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: Oct 06 04:28:46.678391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:46.733216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 755/1497] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:28:46 2022] POST /identity/v3/projects => generated 377 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:28:46.737439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:46.738990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:46.773911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed1f562-0758-4f73-9d7b-2d8d4e93d90f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:46.865492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:46.866082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:46.866353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:46.866626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:46.867257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:46.868249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:46.868249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:28:46.868249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:28:46.869446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:46.910493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 743/1498] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:28:46 2022] POST /identity/v3/users => generated 289 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:28:46.914585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:46.916196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:46.983656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7ec841-02e7-4900-a8d0-f48c71fb0769 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:47.066904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:47.083813 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:47.084463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:47.084743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:47.085022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:47.085799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:47.086885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING py.warnings [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:47.086885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: warnings.warn(msg) Oct 06 04:28:47.086885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: Oct 06 04:28:47.087800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:47.100634 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 756/1499] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:28:46 2022] GET /identity/v3/roles => generated 1420 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:28:47.104737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:47.106814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:47.118310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c499137-4aeb-4045-9f79-f335d303d87c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:47.143733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:47.212585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819fee0-2841-419b-a041-1908b9e5ed57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:47.213106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819fee0-2841-419b-a041-1908b9e5ed57 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:47.213375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819fee0-2841-419b-a041-1908b9e5ed57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:47.213638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819fee0-2841-419b-a041-1908b9e5ed57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:47.214268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819fee0-2841-419b-a041-1908b9e5ed57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:47.215305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-2819fee0-2841-419b-a041-1908b9e5ed57 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:47.215305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:28:47.215305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:28:47.216184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819fee0-2841-419b-a041-1908b9e5ed57 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:47.228772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 744/1500] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:28:47 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:28:47.233417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:47.234940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:47.268858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8a9cb7-c0f9-4116-a1ae-9d44228f2f77 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:47.360852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59df4f9d-4bd5-469a-87f0-538c6e24d6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:47.361489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59df4f9d-4bd5-469a-87f0-538c6e24d6cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:47.361759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59df4f9d-4bd5-469a-87f0-538c6e24d6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:47.362025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59df4f9d-4bd5-469a-87f0-538c6e24d6cd admin admin] PATH_INFO: `/v3/projects/574d293053214c6196a2299b56560dd6/users/0b9eac9472e6444b9b13b1d03b292ec8/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:47.384554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59df4f9d-4bd5-469a-87f0-538c6e24d6cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=574d293053214c6196a2299b56560dd6, user_id=0b9eac9472e6444b9b13b1d03b292ec8, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:47.385582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING py.warnings [None req-59df4f9d-4bd5-469a-87f0-538c6e24d6cd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:47.385582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: warnings.warn(msg) Oct 06 04:28:47.385582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: Oct 06 04:28:47.387789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59df4f9d-4bd5-469a-87f0-538c6e24d6cd admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:47.413527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 757/1501] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:28:47 2022] PUT /identity/v3/projects/574d293053214c6196a2299b56560dd6/users/0b9eac9472e6444b9b13b1d03b292ec8/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:28:47.426050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7b93c0-8a6b-4c7c-bc56-7c2c8edbbdc6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:47.426336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7b93c0-8a6b-4c7c-bc56-7c2c8edbbdc6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:47.426609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7b93c0-8a6b-4c7c-bc56-7c2c8edbbdc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:47.482901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1b7b93c0-8a6b-4c7c-bc56-7c2c8edbbdc6 None None] MFA Rules not processed for user `0b9eac9472e6444b9b13b1d03b292ec8`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:28:47.484044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1b7b93c0-8a6b-4c7c-bc56-7c2c8edbbdc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:47.559245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 745/1502] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:28:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:28:47.572356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f908b41-6886-47f7-a734-783d9f59ed73 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:47.572638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f908b41-6886-47f7-a734-783d9f59ed73 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:47.572911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f908b41-6886-47f7-a734-783d9f59ed73 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:47.574380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 758/1503] 10.222.0.33 () {54 vars in 963 bytes} [Thu Oct 6 04:28:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:28:47.579086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e610cd54-3692-4871-aa5a-2590561b7de9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:47.579386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e610cd54-3692-4871-aa5a-2590561b7de9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:47.579643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e610cd54-3692-4871-aa5a-2590561b7de9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:47.612065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:47.656418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e610cd54-3692-4871-aa5a-2590561b7de9 None None] MFA Rules not processed for user `61d1b78e81db4dbe987d788fd398e89d`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:28:47.657596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e610cd54-3692-4871-aa5a-2590561b7de9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:47.732325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 746/1504] 10.222.0.33 () {58 vars in 1060 bytes} [Thu Oct 6 04:28:47 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:28:47.742095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f908b41-6886-47f7-a734-783d9f59ed73 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:47.743592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1f908b41-6886-47f7-a734-783d9f59ed73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:47.792582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:47.846785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f908b41-6886-47f7-a734-783d9f59ed73 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:47.919455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:47.920045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:47.920322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:47.920589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:47.922158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:48.019617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:48.020994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:48.024442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:48.039938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 759/1505] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:28:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:28:48.086168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312381c5-cb51-4e4c-a05f-edf060925c57 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:48.086456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312381c5-cb51-4e4c-a05f-edf060925c57 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:48.086731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312381c5-cb51-4e4c-a05f-edf060925c57 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:48.088036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 747/1506] 10.222.0.33 () {52 vars in 830 bytes} [Thu Oct 6 04:28:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:28:48.092454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:48.093657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:48.194363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f419445-0215-4d9e-8ed5-8c32e27d3a40 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:48.292315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:48.294093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:48.294093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:48.294093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] PATH_INFO: `/v3/users/0b9eac9472e6444b9b13b1d03b292ec8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:48.306774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b9eac9472e6444b9b13b1d03b292ec8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:48.307816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING py.warnings [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:48.307816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: warnings.warn(msg) Oct 06 04:28:48.307816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: Oct 06 04:28:48.309121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:48.318665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:48.376855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b9eac9472e6444b9b13b1d03b292ec8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:48.377435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b9eac9472e6444b9b13b1d03b292ec8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:48.391688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b9eac9472e6444b9b13b1d03b292ec8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:48.414257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b9eac9472e6444b9b13b1d03b292ec8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:48.414704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b9eac9472e6444b9b13b1d03b292ec8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:48.429155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 760/1507] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:28:48 2022] DELETE /identity/v3/users/0b9eac9472e6444b9b13b1d03b292ec8 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:28:48.433571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:48.434205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:48.434205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:48.491792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:28:48.493002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:48.569295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:48.581789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 748/1508] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:28:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:28:48.591289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:48.592534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:48.692053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434e7bba-c8cd-4f1b-8219-888deceab36f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:48.747641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:48.768555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89a3632-7052-4ca9-9492-7c683f27d4a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:48.769162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89a3632-7052-4ca9-9492-7c683f27d4a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:48.769436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89a3632-7052-4ca9-9492-7c683f27d4a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:48.769703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89a3632-7052-4ca9-9492-7c683f27d4a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:48.771320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f89a3632-7052-4ca9-9492-7c683f27d4a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:48.880528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89a3632-7052-4ca9-9492-7c683f27d4a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:48.881876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89a3632-7052-4ca9-9492-7c683f27d4a1 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:48.900421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 761/1509] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:28:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:28:49.255999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:28:49.257303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:28:49.269063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881e650-d465-40dc-8a14-b96ba00685c7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:28:49.368184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:28:49.368810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:28:49.369087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:28:49.369357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] PATH_INFO: `/v3/projects/574d293053214c6196a2299b56560dd6` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:28:49.379934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] RBAC: Authorizing `identity:delete_project(project_id=574d293053214c6196a2299b56560dd6)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:28:49.380958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:28:49.380958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:28:49.380958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:28:49.382155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:28:49.441462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '574d293053214c6196a2299b56560dd6', 'request_id': 'req-e785835c-fdac-4ab2-8da1-9cdd2b421831'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:49.447312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '574d293053214c6196a2299b56560dd6', 'request_id': 'req-e785835c-fdac-4ab2-8da1-9cdd2b421831'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:49.460700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '574d293053214c6196a2299b56560dd6', 'request_id': 'req-e785835c-fdac-4ab2-8da1-9cdd2b421831'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:49.486812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] The token cache is being invalidate because project 574d293053214c6196a2299b56560dd6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:28:49.487130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:49.487646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e785835c-fdac-4ab2-8da1-9cdd2b421831 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:28:49.488939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 749/1510] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:28:49 2022] DELETE /identity/v3/projects/574d293053214c6196a2299b56560dd6 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:28:58.898196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=62 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:28:59.484844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:30:14.655437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84714d0d-77bb-45f3-9ecd-edf3892aa028 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:14.656036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84714d0d-77bb-45f3-9ecd-edf3892aa028 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:14.656036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84714d0d-77bb-45f3-9ecd-edf3892aa028 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:14.657314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 762/1511] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:30:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:14.662321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffe20a-b2b7-4924-8270-6ba9a4d4e575 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:14.662763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffe20a-b2b7-4924-8270-6ba9a4d4e575 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:14.662878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffe20a-b2b7-4924-8270-6ba9a4d4e575 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:14.744335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-0dffe20a-b2b7-4924-8270-6ba9a4d4e575 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:14.745537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0dffe20a-b2b7-4924-8270-6ba9a4d4e575 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:14.848153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 750/1512] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:30:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:15.007155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84714d0d-77bb-45f3-9ecd-edf3892aa028 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:15.009209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-84714d0d-77bb-45f3-9ecd-edf3892aa028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:15.127324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84714d0d-77bb-45f3-9ecd-edf3892aa028 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:15.133010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8f6525-c1a4-46cf-ab40-60cb9c1a8659 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:15.133618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8f6525-c1a4-46cf-ab40-60cb9c1a8659 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:15.133896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8f6525-c1a4-46cf-ab40-60cb9c1a8659 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:15.134167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8f6525-c1a4-46cf-ab40-60cb9c1a8659 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:15.142497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8f6525-c1a4-46cf-ab40-60cb9c1a8659 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:15.144306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8f6525-c1a4-46cf-ab40-60cb9c1a8659 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:15.154480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 763/1513] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:16.243083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93305f01-4728-463b-9975-4ce436121812 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:16.243592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93305f01-4728-463b-9975-4ce436121812 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:16.243715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93305f01-4728-463b-9975-4ce436121812 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:16.244966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 751/1514] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:30:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:16.251194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026aebb2-b98d-44b6-aef7-5025640bd56b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:16.251613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026aebb2-b98d-44b6-aef7-5025640bd56b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:16.251743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026aebb2-b98d-44b6-aef7-5025640bd56b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:16.271417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-026aebb2-b98d-44b6-aef7-5025640bd56b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:16.272597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-026aebb2-b98d-44b6-aef7-5025640bd56b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:16.281111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 764/1515] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:30:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:16.431354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93305f01-4728-463b-9975-4ce436121812 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:16.445952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93305f01-4728-463b-9975-4ce436121812 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:16.452036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df90e83-ca43-40a8-968e-af2f310e425d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:16.452627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df90e83-ca43-40a8-968e-af2f310e425d service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:16.452904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df90e83-ca43-40a8-968e-af2f310e425d service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:16.453231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df90e83-ca43-40a8-968e-af2f310e425d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:16.461313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df90e83-ca43-40a8-968e-af2f310e425d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:16.462889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df90e83-ca43-40a8-968e-af2f310e425d service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:16.472219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 752/1516] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:16.645482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80e76eb-dd18-48f2-a7ac-e559cc1e0fd0 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:16.645768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80e76eb-dd18-48f2-a7ac-e559cc1e0fd0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:16.646040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80e76eb-dd18-48f2-a7ac-e559cc1e0fd0 None None] PATH_INFO: `/v3` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:16.647344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 765/1517] 10.222.0.25 () {54 vars in 925 bytes} [Thu Oct 6 04:30:16 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:30:16.654345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedccc83-550c-47da-a828-6a9eec9f0c18 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:16.654628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedccc83-550c-47da-a828-6a9eec9f0c18 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:16.654902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedccc83-550c-47da-a828-6a9eec9f0c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:16.713996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-cedccc83-550c-47da-a828-6a9eec9f0c18 None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:16.718487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cedccc83-550c-47da-a828-6a9eec9f0c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:16.733226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:30:16.787584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 753/1518] 10.222.0.25 () {58 vars in 1015 bytes} [Thu Oct 6 04:30:16 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:25.408712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8880ae-9842-48cd-a67a-3903b6ceaca3 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:25.409189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8880ae-9842-48cd-a67a-3903b6ceaca3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:25.409292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8880ae-9842-48cd-a67a-3903b6ceaca3 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:25.410570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 766/1519] 10.222.0.33 () {54 vars in 961 bytes} [Thu Oct 6 04:30:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:25.415772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9d692a-3f37-429d-97fb-8a5ac42328ad None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:25.416205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9d692a-3f37-429d-97fb-8a5ac42328ad None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:25.416358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9d692a-3f37-429d-97fb-8a5ac42328ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:25.436835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fc9d692a-3f37-429d-97fb-8a5ac42328ad None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:25.438045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fc9d692a-3f37-429d-97fb-8a5ac42328ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:25.446782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 754/1520] 10.222.0.33 () {58 vars in 1058 bytes} [Thu Oct 6 04:30:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:25.454349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8880ae-9842-48cd-a67a-3903b6ceaca3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:25.468863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8880ae-9842-48cd-a67a-3903b6ceaca3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:25.473849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a73a6f-96f5-4f0a-8aae-231aae9446f2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:25.474437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a73a6f-96f5-4f0a-8aae-231aae9446f2 service placement] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:25.474710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a73a6f-96f5-4f0a-8aae-231aae9446f2 service placement] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:25.475003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a73a6f-96f5-4f0a-8aae-231aae9446f2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:25.483583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a73a6f-96f5-4f0a-8aae-231aae9446f2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:25.485136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a73a6f-96f5-4f0a-8aae-231aae9446f2 service placement] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:25.495059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 767/1521] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:27.996638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2519585e-be19-4127-93cd-c67666d96d2d None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:27.997202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2519585e-be19-4127-93cd-c67666d96d2d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:27.997202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2519585e-be19-4127-93cd-c67666d96d2d None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:27.998516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 755/1522] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:30:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:28.004610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cec511-760e-4126-95cd-235431e5a2f1 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:28.005072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cec511-760e-4126-95cd-235431e5a2f1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:28.005169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cec511-760e-4126-95cd-235431e5a2f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:28.024819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-78cec511-760e-4126-95cd-235431e5a2f1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:28.026008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-78cec511-760e-4126-95cd-235431e5a2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:28.033788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 768/1523] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:30:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:28.187405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2519585e-be19-4127-93cd-c67666d96d2d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:28.202396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2519585e-be19-4127-93cd-c67666d96d2d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:28.207783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8d8957-531a-44ca-80ae-a66997362498 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:28.208366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d8957-531a-44ca-80ae-a66997362498 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:28.208672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d8957-531a-44ca-80ae-a66997362498 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:28.208944 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d8957-531a-44ca-80ae-a66997362498 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:28.217288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8d8957-531a-44ca-80ae-a66997362498 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:28.218872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8d8957-531a-44ca-80ae-a66997362498 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:28.228348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 756/1524] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:29.302713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b8410a-eb15-41fa-8545-770d8b16740e None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:29.303336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b8410a-eb15-41fa-8545-770d8b16740e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:29.303336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b8410a-eb15-41fa-8545-770d8b16740e None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:29.304615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 769/1525] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 6 04:30:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:29.311457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d3696-db0c-46ae-9bd4-629248c75dda None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:29.311879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d3696-db0c-46ae-9bd4-629248c75dda None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:29.311989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621d3696-db0c-46ae-9bd4-629248c75dda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:29.333269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-621d3696-db0c-46ae-9bd4-629248c75dda None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:29.334460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-621d3696-db0c-46ae-9bd4-629248c75dda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:29.342458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 757/1526] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 6 04:30:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:29.494444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b8410a-eb15-41fa-8545-770d8b16740e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:29.508951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b8410a-eb15-41fa-8545-770d8b16740e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:29.514831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adafe690-5444-486b-9a38-0db65d4e78f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:29.515448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adafe690-5444-486b-9a38-0db65d4e78f9 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:29.515717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adafe690-5444-486b-9a38-0db65d4e78f9 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:29.515983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adafe690-5444-486b-9a38-0db65d4e78f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:29.523795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adafe690-5444-486b-9a38-0db65d4e78f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:29.525338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adafe690-5444-486b-9a38-0db65d4e78f9 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:29.534498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 770/1527] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:30:33.402860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e22a53-d414-46f2-bc22-28198cbd9ff2 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:33.403452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e22a53-d414-46f2-bc22-28198cbd9ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:33.403452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e22a53-d414-46f2-bc22-28198cbd9ff2 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:33.404889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 758/1528] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:30:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:33.410218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d6ec86-2278-4d15-bc49-f3d0b2945be6 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:33.410624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d6ec86-2278-4d15-bc49-f3d0b2945be6 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:33.410753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d6ec86-2278-4d15-bc49-f3d0b2945be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:33.430361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f2d6ec86-2278-4d15-bc49-f3d0b2945be6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:33.431580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f2d6ec86-2278-4d15-bc49-f3d0b2945be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:33.440026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 771/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:30:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:30:33.725990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e22a53-d414-46f2-bc22-28198cbd9ff2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:33.727771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f8e22a53-d414-46f2-bc22-28198cbd9ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:33.805904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e22a53-d414-46f2-bc22-28198cbd9ff2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:33.809991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0776e0c-45bc-446d-9e8d-624f9f680b44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:33.810584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0776e0c-45bc-446d-9e8d-624f9f680b44 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:33.810858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0776e0c-45bc-446d-9e8d-624f9f680b44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:33.811157 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0776e0c-45bc-446d-9e8d-624f9f680b44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:33.819896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0776e0c-45bc-446d-9e8d-624f9f680b44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:33.821308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0776e0c-45bc-446d-9e8d-624f9f680b44 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:33.830753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 759/1530] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:35.984319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c073cbde-8d7f-421a-af28-7f014629c200 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:35.984833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c073cbde-8d7f-421a-af28-7f014629c200 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:35.984833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c073cbde-8d7f-421a-af28-7f014629c200 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:35.986175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 772/1531] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:30:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:35.991690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23982344-bded-46dc-8763-18987ec91f80 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:35.992118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23982344-bded-46dc-8763-18987ec91f80 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:35.992225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23982344-bded-46dc-8763-18987ec91f80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:36.011804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-23982344-bded-46dc-8763-18987ec91f80 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:36.012997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-23982344-bded-46dc-8763-18987ec91f80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:36.020731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 760/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:30:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:36.307773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c073cbde-8d7f-421a-af28-7f014629c200 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:36.318480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c073cbde-8d7f-421a-af28-7f014629c200 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:36.321946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70401e1a-ca9d-49c7-b2ce-f52133f24439 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:36.322527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70401e1a-ca9d-49c7-b2ce-f52133f24439 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:36.322799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70401e1a-ca9d-49c7-b2ce-f52133f24439 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:36.323101 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70401e1a-ca9d-49c7-b2ce-f52133f24439 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:36.332258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70401e1a-ca9d-49c7-b2ce-f52133f24439 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:36.333652 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70401e1a-ca9d-49c7-b2ce-f52133f24439 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:36.343721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 773/1533] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:39.964041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e58804-7a6c-460d-b66a-51b841119d7c None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:39.964639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e58804-7a6c-460d-b66a-51b841119d7c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:39.964639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e58804-7a6c-460d-b66a-51b841119d7c None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:39.966104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 761/1534] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:30:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:39.971413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6211c55-7c37-4886-8e2a-7b1603ca25f8 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:39.971887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6211c55-7c37-4886-8e2a-7b1603ca25f8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:39.971983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6211c55-7c37-4886-8e2a-7b1603ca25f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:39.979594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=81 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:30:39.992800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b6211c55-7c37-4886-8e2a-7b1603ca25f8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:39.994003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b6211c55-7c37-4886-8e2a-7b1603ca25f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:40.002562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 774/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:30:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:40.313289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e58804-7a6c-460d-b66a-51b841119d7c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:40.314973 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-89e58804-7a6c-460d-b66a-51b841119d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:40.383081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:30:40.397765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e58804-7a6c-460d-b66a-51b841119d7c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:40.401401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f31879-011b-43ff-8b8d-c17fd4719530 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:40.401949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f31879-011b-43ff-8b8d-c17fd4719530 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:40.402502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f31879-011b-43ff-8b8d-c17fd4719530 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:40.402774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f31879-011b-43ff-8b8d-c17fd4719530 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:40.412480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f31879-011b-43ff-8b8d-c17fd4719530 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:40.413908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f31879-011b-43ff-8b8d-c17fd4719530 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:40.423708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 762/1536] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:42.594275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d9759-86f9-476b-ba71-2549a6255deb None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:42.594800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d9759-86f9-476b-ba71-2549a6255deb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:42.594800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d9759-86f9-476b-ba71-2549a6255deb None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:42.596189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 775/1537] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 6 04:30:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:30:42.601733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1d0fd-ce70-42f2-bdf4-89b5d4dc8851 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:42.602198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1d0fd-ce70-42f2-bdf4-89b5d4dc8851 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:42.602305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1d0fd-ce70-42f2-bdf4-89b5d4dc8851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:42.622948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a8f1d0fd-ce70-42f2-bdf4-89b5d4dc8851 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:30:42.624171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a8f1d0fd-ce70-42f2-bdf4-89b5d4dc8851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:42.632275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 763/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 6 04:30:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:30:42.938104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d9759-86f9-476b-ba71-2549a6255deb None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:30:42.939976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a74d9759-86f9-476b-ba71-2549a6255deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:30:42.953329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d9759-86f9-476b-ba71-2549a6255deb None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:30:42.956941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c49b01-5342-4edb-b61a-1706e7f88fba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:30:42.957527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c49b01-5342-4edb-b61a-1706e7f88fba service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:30:42.957799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c49b01-5342-4edb-b61a-1706e7f88fba service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:30:42.958065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c49b01-5342-4edb-b61a-1706e7f88fba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:30:42.970433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c49b01-5342-4edb-b61a-1706e7f88fba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:30:42.971971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c49b01-5342-4edb-b61a-1706e7f88fba service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:30:42.982421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 776/1539] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:30:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:30:52.617630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=11 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:30:52.977543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=8 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:06.630342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dd8285-c642-4fba-952b-e4be9baf82ae None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.631039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dd8285-c642-4fba-952b-e4be9baf82ae None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.631039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dd8285-c642-4fba-952b-e4be9baf82ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.658060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-39dd8285-c642-4fba-952b-e4be9baf82ae None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:06.659334 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-39dd8285-c642-4fba-952b-e4be9baf82ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:06.672639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 764/1540] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:06.678439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49d343-f0b2-441e-ae1e-9a9d404bbad9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.679043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49d343-f0b2-441e-ae1e-9a9d404bbad9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.679043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c49d343-f0b2-441e-ae1e-9a9d404bbad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.684413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1b7288-1752-4d54-ac60-130b29726e0e None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.684726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1b7288-1752-4d54-ac60-130b29726e0e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.684966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1b7288-1752-4d54-ac60-130b29726e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.718029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8c49d343-f0b2-441e-ae1e-9a9d404bbad9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:06.719279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8c49d343-f0b2-441e-ae1e-9a9d404bbad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:06.721549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ec1b7288-1752-4d54-ac60-130b29726e0e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:06.722181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec1b7288-1752-4d54-ac60-130b29726e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:06.736535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 765/1541] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:06.738704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41af4260-00c6-4c93-a14d-795570a47b00 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.743034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 777/1542] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:06.743034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49d343-f0b2-441e-ae1e-9a9d404bbad9 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:06.744032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41af4260-00c6-4c93-a14d-795570a47b00 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.744620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41af4260-00c6-4c93-a14d-795570a47b00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.760007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c49d343-f0b2-441e-ae1e-9a9d404bbad9 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:06.779017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-41af4260-00c6-4c93-a14d-795570a47b00 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:06.780623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-41af4260-00c6-4c93-a14d-795570a47b00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:06.782309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:06.783069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.783376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.783646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.784297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:06.784595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:06.785592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING py.warnings [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:32:06.785592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: warnings.warn(msg) Oct 06 04:32:06.785592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: Oct 06 04:32:06.787706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ec022-9fd0-481e-b60a-0987d362855c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:06.800291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 766/1543] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:06.801918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c90200-a1c3-46e4-8420-26400ee81485 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.802488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c90200-a1c3-46e4-8420-26400ee81485 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.803012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c90200-a1c3-46e4-8420-26400ee81485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.812002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 778/1544] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:32:06.825969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc990b5-2dae-40c6-829b-1d31d2378cee None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.826315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc990b5-2dae-40c6-829b-1d31d2378cee None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.826563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc990b5-2dae-40c6-829b-1d31d2378cee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.837311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-01c90200-a1c3-46e4-8420-26400ee81485 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:06.839858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-01c90200-a1c3-46e4-8420-26400ee81485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:06.856765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 767/1545] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:06.857503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c90200-a1c3-46e4-8420-26400ee81485 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:06.865627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-cbc990b5-2dae-40c6-829b-1d31d2378cee None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:06.867003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-cbc990b5-2dae-40c6-829b-1d31d2378cee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:06.874750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c90200-a1c3-46e4-8420-26400ee81485 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:06.881856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 779/1546] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:06.883656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc990b5-2dae-40c6-829b-1d31d2378cee None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:06.888104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:06.888730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.889130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.889327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.889910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:06.891032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:32:06.891032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:32:06.891032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:32:06.892237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:06.902693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc990b5-2dae-40c6-829b-1d31d2378cee None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:06.931317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282dda6-4650-4507-94fe-9c9056a23f3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:06.931935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282dda6-4650-4507-94fe-9c9056a23f3b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.932707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282dda6-4650-4507-94fe-9c9056a23f3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.933385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282dda6-4650-4507-94fe-9c9056a23f3b admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.933682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282dda6-4650-4507-94fe-9c9056a23f3b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:06.933990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282dda6-4650-4507-94fe-9c9056a23f3b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:06.935617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282dda6-4650-4507-94fe-9c9056a23f3b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:06.969528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 780/1547] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:06.971244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f1903-de83-4a09-b863-dfd2d0d54fc2 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:06.971519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f1903-de83-4a09-b863-dfd2d0d54fc2 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:06.971791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f1903-de83-4a09-b863-dfd2d0d54fc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:06.989012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 768/1548] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/projects => generated 373 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:06.989859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.017494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-792f1903-de83-4a09-b863-dfd2d0d54fc2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.018732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-792f1903-de83-4a09-b863-dfd2d0d54fc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.068443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b690314-65f9-4c88-8f88-a73d1f356e2b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.073520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.073821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.074118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.074363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.075068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:07.075373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.076985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.082741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 781/1549] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.084087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154eae07-3ec1-4fc5-adea-351bfedb198c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.084371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154eae07-3ec1-4fc5-adea-351bfedb198c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.084641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154eae07-3ec1-4fc5-adea-351bfedb198c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.091052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 769/1550] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:07.092119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.103173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ae47da-e09b-4055-ae53-593b073d6572 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.107886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-154eae07-3ec1-4fc5-adea-351bfedb198c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.108536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56b3d65-9fbf-4ebb-8d6e-73913ec426a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.109118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b3d65-9fbf-4ebb-8d6e-73913ec426a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.109231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-154eae07-3ec1-4fc5-adea-351bfedb198c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.109392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b3d65-9fbf-4ebb-8d6e-73913ec426a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.109659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b3d65-9fbf-4ebb-8d6e-73913ec426a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.110278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56b3d65-9fbf-4ebb-8d6e-73913ec426a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.112290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56b3d65-9fbf-4ebb-8d6e-73913ec426a5 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.117504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 782/1551] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.118594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154eae07-3ec1-4fc5-adea-351bfedb198c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.130888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154eae07-3ec1-4fc5-adea-351bfedb198c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.136483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52a3b86-8928-4aeb-804d-28bd942e523d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.137041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52a3b86-8928-4aeb-804d-28bd942e523d admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.137316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52a3b86-8928-4aeb-804d-28bd942e523d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.137579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52a3b86-8928-4aeb-804d-28bd942e523d admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.138192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52a3b86-8928-4aeb-804d-28bd942e523d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.139216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING py.warnings [None req-c52a3b86-8928-4aeb-804d-28bd942e523d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:32:07.139216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: warnings.warn(msg) Oct 06 04:32:07.139216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: Oct 06 04:32:07.140399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52a3b86-8928-4aeb-804d-28bd942e523d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.141401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 770/1552] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/projects => generated 381 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.142690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d014c5-d963-4edd-9186-abd6dc891e08 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.142915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d014c5-d963-4edd-9186-abd6dc891e08 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.143205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d014c5-d963-4edd-9186-abd6dc891e08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.161855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 783/1553] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/users => generated 287 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.163132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbed135e-c2ca-46bf-aad6-e27f77057633 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.163410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbed135e-c2ca-46bf-aad6-e27f77057633 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.163675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbed135e-c2ca-46bf-aad6-e27f77057633 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.164124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-41d014c5-d963-4edd-9186-abd6dc891e08 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.165571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-41d014c5-d963-4edd-9186-abd6dc891e08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.184219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-fbed135e-c2ca-46bf-aad6-e27f77057633 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.185694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fbed135e-c2ca-46bf-aad6-e27f77057633 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.228515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 771/1554] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.230062 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d7007-8b0b-40a1-bee9-1c50463d4717 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.230341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d7007-8b0b-40a1-bee9-1c50463d4717 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.230611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d7007-8b0b-40a1-bee9-1c50463d4717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.248871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 784/1555] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.250209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c70379-8cba-4296-85f7-18027739d794 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.250525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c70379-8cba-4296-85f7-18027739d794 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.250786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c70379-8cba-4296-85f7-18027739d794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.253620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-da5d7007-8b0b-40a1-bee9-1c50463d4717 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.254980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-da5d7007-8b0b-40a1-bee9-1c50463d4717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.264023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 772/1556] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.264798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d7007-8b0b-40a1-bee9-1c50463d4717 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.274351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d7007-8b0b-40a1-bee9-1c50463d4717 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.278053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-20c70379-8cba-4296-85f7-18027739d794 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.279321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-20c70379-8cba-4296-85f7-18027739d794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.279727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.280313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.280584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.280848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.281491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.283428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.287392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 785/1557] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.288558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8b75f-3a85-486d-888c-d5f61b16a2de None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.288842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8b75f-3a85-486d-888c-d5f61b16a2de None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.289116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8b75f-3a85-486d-888c-d5f61b16a2de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.292956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:07.311169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-22d8b75f-3a85-486d-888c-d5f61b16a2de None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.312366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-22d8b75f-3a85-486d-888c-d5f61b16a2de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.319122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 773/1558] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.319122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.327210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 786/1559] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.327210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8b75f-3a85-486d-888c-d5f61b16a2de None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.334152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8b75f-3a85-486d-888c-d5f61b16a2de None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.355405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:07.381055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3b9e22-88ca-4390-884f-61a1b4091a4c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.386661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d361bc-53fb-4745-9a57-c4be18a4d672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.387219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d361bc-53fb-4745-9a57-c4be18a4d672 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.387495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d361bc-53fb-4745-9a57-c4be18a4d672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.387771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d361bc-53fb-4745-9a57-c4be18a4d672 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.388403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d361bc-53fb-4745-9a57-c4be18a4d672 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.390322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d361bc-53fb-4745-9a57-c4be18a4d672 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.395874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7915bc-68d4-493e-9e1b-152068e32a72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.396469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7915bc-68d4-493e-9e1b-152068e32a72 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.396743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7915bc-68d4-493e-9e1b-152068e32a72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.397011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7915bc-68d4-493e-9e1b-152068e32a72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.397644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7915bc-68d4-493e-9e1b-152068e32a72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.399328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7915bc-68d4-493e-9e1b-152068e32a72 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.413259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 787/1560] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:07.418425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 774/1561] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/users => generated 291 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.418615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57cec57-d510-4da1-b354-758bc4654469 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.418615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57cec57-d510-4da1-b354-758bc4654469 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.418615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57cec57-d510-4da1-b354-758bc4654469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.419520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b25ee2-0144-400a-b22f-52239a42f8b0 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.419520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b25ee2-0144-400a-b22f-52239a42f8b0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.419520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b25ee2-0144-400a-b22f-52239a42f8b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.437879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a3b25ee2-0144-400a-b22f-52239a42f8b0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.438819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e57cec57-d510-4da1-b354-758bc4654469 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.439125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a3b25ee2-0144-400a-b22f-52239a42f8b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.440544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e57cec57-d510-4da1-b354-758bc4654469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.447003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 775/1562] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.448428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f13081-eef8-4a20-a01f-fb6d6da0478f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.448749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f13081-eef8-4a20-a01f-fb6d6da0478f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.449021 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f13081-eef8-4a20-a01f-fb6d6da0478f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.453826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 788/1563] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.454666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57cec57-d510-4da1-b354-758bc4654469 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.465287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57cec57-d510-4da1-b354-758bc4654469 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.469410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-49f13081-eef8-4a20-a01f-fb6d6da0478f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:07.469890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.470507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.470618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-49f13081-eef8-4a20-a01f-fb6d6da0478f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:07.470776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.471070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.471699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:07.471994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.473595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.478694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 776/1564] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:07.480131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f13081-eef8-4a20-a01f-fb6d6da0478f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.486574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 789/1565] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:07.487629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.494604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f13081-eef8-4a20-a01f-fb6d6da0478f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.497502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8872a-1ffa-4d87-8d65-38fc42841fe8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.499715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.500337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.500608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.500871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.501523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.502124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.502694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.502985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.503274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.503492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.503897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:07.504195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.505745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.517878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 790/1566] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:07.517878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.528983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec0dc49-98ab-4ecd-b2d8-e8566ed1dfe6 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.535268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 777/1567] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/users => generated 289 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.535268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.535518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.535518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.535518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.535518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.535872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.536869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.543761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a661c293-22ae-478a-8d22-42b4df1140fb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.551150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 791/1568] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:07.551310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.551466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.551572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.551572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.551572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.551572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.551912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.558710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0835b4a-2e79-4068-8139-02b178b075ad admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.562278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 778/1569] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:07.563175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.566461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.567074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.567353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.567622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.568285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:07.568579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.570160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.572690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e407b71-8254-49c1-a0ac-2eb8d676fb36 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.577229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.577804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.578080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.578356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.579004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:07.579309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.580865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.587868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 792/1570] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:07.587868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.593063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 779/1571] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:07.593960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.594516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0829de55-9fa1-441e-afe7-73ae99a3abc3 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.599469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.600602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.600883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.601154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.601915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:07.602222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.603491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecadd16e-4201-490a-9e55-442a227201ca admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.603825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.608656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.609234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.609510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.609779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.610405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.612316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.616604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 793/1572] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:07.616604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.627339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e486b5-8aa8-4dd6-b275-429c93ec20e9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.631974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.632545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.632822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.633094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.633812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.636159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.648384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 780/1573] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.649144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.665752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 794/1574] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/projects => generated 377 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.666718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.725013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461e3b88-0385-48f4-8951-9f9138cf2dd5 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.728732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:07.735086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f950aa-5102-4cad-a4a5-449913de7466 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.740026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.740684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.740959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.741232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] PATH_INFO: `/v3/projects/0b73eb6e32754817806e3b97b0d1b402/users/2c6627dc1a8c4a7197c3a1d3f75decc5/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.757733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b73eb6e32754817806e3b97b0d1b402, user_id=2c6627dc1a8c4a7197c3a1d3f75decc5, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.760964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.773538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 795/1575] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:07 2022] PUT /identity/v3/projects/0b73eb6e32754817806e3b97b0d1b402/users/2c6627dc1a8c4a7197c3a1d3f75decc5/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:07.776069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.786820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.787464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.787757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.788012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.788659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.790266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.790399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:07.800939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 781/1576] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:07.801812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.811688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447f0e7-1345-4657-9968-83715105c925 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.841784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d8d27-ac77-4e86-a626-56860bfc15ef admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.846395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.847017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.847293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.847563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.848195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.849779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.860643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 796/1577] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:07 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:07.861772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.870941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.871443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1dbffb-f080-460d-8c8f-3deee56052cc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.871591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.871848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.872118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.872748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.874647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.876058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.876631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.876914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.877182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.877795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.879693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.905110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 782/1578] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/projects => generated 391 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.905925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.910561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 797/1579] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/projects => generated 389 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:07.911431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:07.971999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6949563c-9fe6-4392-adfd-3bc6fe2f801b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.974795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b36904d-201e-492b-a900-ff9bed2a94bc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:07.976645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b678fb41-cf13-436a-9cff-1a1ba7944db4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.977236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b678fb41-cf13-436a-9cff-1a1ba7944db4 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.977514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b678fb41-cf13-436a-9cff-1a1ba7944db4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.977781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b678fb41-cf13-436a-9cff-1a1ba7944db4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.978399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b678fb41-cf13-436a-9cff-1a1ba7944db4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.979504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:07.980095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:07.980372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:07.980490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b678fb41-cf13-436a-9cff-1a1ba7944db4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:07.980646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:07.981266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:07.983178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.004566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 798/1580] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:08.009772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 783/1581] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:07 2022] POST /identity/v3/projects => generated 373 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:08.009772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17e5b3-94dc-417e-b807-a5347f3523c8 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.009772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17e5b3-94dc-417e-b807-a5347f3523c8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.009772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf17e5b3-94dc-417e-b807-a5347f3523c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.010166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.019339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f991cb2-cdfb-4e50-b0d8-28e5d1fc2a76 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.034315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:08.037641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-bf17e5b3-94dc-417e-b807-a5347f3523c8 None None] MFA Rules not processed for user `2c6627dc1a8c4a7197c3a1d3f75decc5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:08.039782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-bf17e5b3-94dc-417e-b807-a5347f3523c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:08.077361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.077963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.078244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.078515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.079174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.081149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.102091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 799/1582] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:08.102897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.109346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:08.110033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 784/1583] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:08.110910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf17e5b3-94dc-417e-b807-a5347f3523c8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.113603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8216f1b8-f200-4db3-9db0-c889faca2693 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.119590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.121437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.122110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.122591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.123919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.127487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.133201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf17e5b3-94dc-417e-b807-a5347f3523c8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.143536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.144670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.145127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.145739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] PATH_INFO: `/v3/projects/74ed4afcaeed46a3ae1f8ba2c62f1d3f/users/1847fad78ead4f41b5705b2c555dacee/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.146129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 800/1584] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.147373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.162698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57026443-c3a3-48bb-b8de-3657161b1513 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.170157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.171000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.171440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.171893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.172915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.174868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74ed4afcaeed46a3ae1f8ba2c62f1d3f, user_id=1847fad78ead4f41b5705b2c555dacee, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.176663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:32:08.176663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:32:08.176663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:32:08.177084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.180150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.199287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 785/1585] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:08 2022] PUT /identity/v3/projects/74ed4afcaeed46a3ae1f8ba2c62f1d3f/users/1847fad78ead4f41b5705b2c555dacee/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:08.200586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.203336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 801/1586] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/users => generated 296 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:08.204308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.214745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb68940-b046-4f77-acba-d82565db8acf admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.273973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.274574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.274855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.275161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.275806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.278342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.281838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0744e3-b6c2-41ce-b0d5-0dcdcaf31247 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.286952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.287584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.287861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.288104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 802/1587] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.288231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.288750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.288953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.290772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.300274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78c8597-10df-458b-8144-853e9a1b0c4f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.305467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c26f25c-76ec-4013-af4c-a1d045f08a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.305976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c26f25c-76ec-4013-af4c-a1d045f08a65 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.306250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c26f25c-76ec-4013-af4c-a1d045f08a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.306647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c26f25c-76ec-4013-af4c-a1d045f08a65 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.307292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c26f25c-76ec-4013-af4c-a1d045f08a65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.309819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c26f25c-76ec-4013-af4c-a1d045f08a65 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.314737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 786/1588] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/users => generated 295 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:08.318801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.328921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad72effd-55a6-4c1b-a7a2-680bae1dab22 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.331515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 803/1589] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/users => generated 287 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:08.332738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4b57e9-e625-4805-b4ac-def07e253f26 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.333031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4b57e9-e625-4805-b4ac-def07e253f26 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.333310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4b57e9-e625-4805-b4ac-def07e253f26 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.333777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.334443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.334675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.334974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.335678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.337275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.339097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 804/1590] 10.222.0.33 () {54 vars in 961 bytes} [Thu Oct 6 04:32:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:32:08.340336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4b57e9-e625-4805-b4ac-def07e253f26 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.348465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 787/1591] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.349338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.352242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4b57e9-e625-4805-b4ac-def07e253f26 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.357651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.358290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.358564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.358829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] PATH_INFO: `/v3/projects/93b9baf342f94f34ae8f649db7198137/users/4f29428b32454dbdb9c644759389abec/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.359811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c67fa0-b54f-44e2-ad97-9260bd1478e0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.364403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8914da-1578-4ee6-8651-67eb1586edd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.364989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8914da-1578-4ee6-8651-67eb1586edd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.365269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8914da-1578-4ee6-8651-67eb1586edd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.365544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8914da-1578-4ee6-8651-67eb1586edd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.366176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8914da-1578-4ee6-8651-67eb1586edd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.368092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8914da-1578-4ee6-8651-67eb1586edd8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.375415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] RBAC: Authorizing `identity:create_grant(project_id=93b9baf342f94f34ae8f649db7198137, user_id=4f29428b32454dbdb9c644759389abec, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.378098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.379713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 788/1592] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.384371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ad7230-d5de-4f9d-a036-dc854a173aef None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.384779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ad7230-d5de-4f9d-a036-dc854a173aef None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.385178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ad7230-d5de-4f9d-a036-dc854a173aef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.392226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 805/1593] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:08 2022] PUT /identity/v3/projects/93b9baf342f94f34ae8f649db7198137/users/4f29428b32454dbdb9c644759389abec/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:08.395954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.413867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f5ad7230-d5de-4f9d-a036-dc854a173aef None None] MFA Rules not processed for user `1847fad78ead4f41b5705b2c555dacee`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:08.415052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f5ad7230-d5de-4f9d-a036-dc854a173aef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:08.451866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:08.461856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f57c79-eba8-4242-ba48-15d333e0f27a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.468460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.469059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.469330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.469598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.470370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.471977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.482024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 806/1594] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.482820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.488181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 789/1595] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:08.489240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ad7230-d5de-4f9d-a036-dc854a173aef None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.492868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8530153e-4768-4e40-b977-f579bf3641d9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.498253 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38557af-6af8-40d3-9cd5-f68fa5043fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.498835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38557af-6af8-40d3-9cd5-f68fa5043fc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.499148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38557af-6af8-40d3-9cd5-f68fa5043fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.499408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38557af-6af8-40d3-9cd5-f68fa5043fc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.499871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ad7230-d5de-4f9d-a036-dc854a173aef None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.500032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38557af-6af8-40d3-9cd5-f68fa5043fc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.501602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38557af-6af8-40d3-9cd5-f68fa5043fc6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.504371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.504957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.505229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.505501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.506133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.507751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.512340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 807/1596] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.513498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66c1b1c-c709-40da-9282-397d09d65a9a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.513776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66c1b1c-c709-40da-9282-397d09d65a9a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.514046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66c1b1c-c709-40da-9282-397d09d65a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.523511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 790/1597] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.523511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.533236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca65ea-e2b7-4fe0-9d94-6e731e63126c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.537982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.538568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.538845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.539146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.539779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.541387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.543227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:08.552024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 791/1598] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.552901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.556903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d66c1b1c-c709-40da-9282-397d09d65a9a None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:08.559274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d66c1b1c-c709-40da-9282-397d09d65a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:08.563024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30c02f8-3c43-42db-aaee-37dcafb6ba61 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.567798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cf8771-281c-4306-9ca9-53480b6cfb17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.568374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cf8771-281c-4306-9ca9-53480b6cfb17 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.568657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cf8771-281c-4306-9ca9-53480b6cfb17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.568929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cf8771-281c-4306-9ca9-53480b6cfb17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.569560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cf8771-281c-4306-9ca9-53480b6cfb17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.571166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cf8771-281c-4306-9ca9-53480b6cfb17 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.581622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 792/1599] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.582833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe92f6b-3c5e-4edb-ad79-bd8532e8518d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.583332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe92f6b-3c5e-4edb-ad79-bd8532e8518d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.583606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe92f6b-3c5e-4edb-ad79-bd8532e8518d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.602727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dbe92f6b-3c5e-4edb-ad79-bd8532e8518d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:08.604656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dbe92f6b-3c5e-4edb-ad79-bd8532e8518d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:08.615808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 793/1600] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:08.615808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe92f6b-3c5e-4edb-ad79-bd8532e8518d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.622935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe92f6b-3c5e-4edb-ad79-bd8532e8518d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.627456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.628100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.628379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.628649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] PATH_INFO: `/v3/projects/653e21693b6f4a3ca8b739f1290df3aa/users/f68d73b5efc74b3380429cf72e506905/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.630292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 808/1601] 10.222.0.33 () {58 vars in 1058 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:08.631131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66c1b1c-c709-40da-9282-397d09d65a9a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.642791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66c1b1c-c709-40da-9282-397d09d65a9a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.643533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=653e21693b6f4a3ca8b739f1290df3aa, user_id=f68d73b5efc74b3380429cf72e506905, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.646188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.647266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.647841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.648115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.648550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.649172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.651844 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.660939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 794/1602] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:08 2022] PUT /identity/v3/projects/653e21693b6f4a3ca8b739f1290df3aa/users/f68d73b5efc74b3380429cf72e506905/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:08.661744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.681187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 809/1603] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/projects => generated 379 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:08.681924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.730451 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa6ff05-bbf4-4f12-aab2-7a0ea361e9dd admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.743730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ffb35d-e1d3-4613-856d-0f533f529e3f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.748629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.749217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.749500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.749751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.750499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.752344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.763058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 810/1604] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.763649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.772727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0b9090-072c-4593-ab18-3037a8cc962e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.779163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.779790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.780085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.780359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] PATH_INFO: `/v3/projects/f53abe2cc7a94ddaadb68b26b57faf69/users/56c86f73f98a4fb2af787534fd47e2ed/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.791212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.792001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.792283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.792558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.793197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.794524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f53abe2cc7a94ddaadb68b26b57faf69, user_id=56c86f73f98a4fb2af787534fd47e2ed, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.794789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.797160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.805713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 795/1605] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/roles => generated 1420 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:08.806807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.814018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 811/1606] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:08 2022] PUT /identity/v3/projects/f53abe2cc7a94ddaadb68b26b57faf69/users/56c86f73f98a4fb2af787534fd47e2ed/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:08.815029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.819803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d973a58-d4ca-4046-a85b-1026cb8a8d20 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.831950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:08.861641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:08.873486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3204e48-4c4d-4802-995d-72c8ffb0fd44 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:08.877299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.877919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.878191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.878457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.879955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe4d8b7-2119-4e3f-9282-88790e5be5e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:08.880599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4d8b7-2119-4e3f-9282-88790e5be5e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.880872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4d8b7-2119-4e3f-9282-88790e5be5e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.881139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe4d8b7-2119-4e3f-9282-88790e5be5e1 admin admin] PATH_INFO: `/v3/projects/0d643fb7b9314f0a9bb8f2c37f06bb2d/users/a58cd041c02d46dcb8acf6adb11ee9de/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.887590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.888999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.896802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe4d8b7-2119-4e3f-9282-88790e5be5e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d643fb7b9314f0a9bb8f2c37f06bb2d, user_id=a58cd041c02d46dcb8acf6adb11ee9de, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:08.898689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 812/1607] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:08.899516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:08.899627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe4d8b7-2119-4e3f-9282-88790e5be5e1 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:08.917615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 796/1608] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:08 2022] PUT /identity/v3/projects/0d643fb7b9314f0a9bb8f2c37f06bb2d/users/a58cd041c02d46dcb8acf6adb11ee9de/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:08.918880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b94063b-526d-47da-9dc1-7d5c309fb51d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:08.919186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b94063b-526d-47da-9dc1-7d5c309fb51d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:08.919552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b94063b-526d-47da-9dc1-7d5c309fb51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:08.941660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1b94063b-526d-47da-9dc1-7d5c309fb51d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:08.942743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1b94063b-526d-47da-9dc1-7d5c309fb51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:08.982879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ec9b5c-2ea0-4298-abd2-156c8a40bf39 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.009798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 797/1609] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:09.011052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b94063b-526d-47da-9dc1-7d5c309fb51d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.029916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b94063b-526d-47da-9dc1-7d5c309fb51d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.034579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.035226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.035502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.035778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.036409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.038359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.063816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 798/1610] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:09 2022] POST /identity/v3/users => generated 296 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:09.064819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.076822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18f09bd-58a0-4488-b439-cc075b7fbea0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.081518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.082281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.082434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.082930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] PATH_INFO: `/v3/projects/bd2a81810fcb4a11abc110f5e90052d4/users/06932e0595d442bdaa2f825384222551/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.095578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a5c4ac-f039-4858-b47c-c616192f245d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.096595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a5c4ac-f039-4858-b47c-c616192f245d service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.096885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a5c4ac-f039-4858-b47c-c616192f245d service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.097156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a5c4ac-f039-4858-b47c-c616192f245d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.100076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd2a81810fcb4a11abc110f5e90052d4, user_id=06932e0595d442bdaa2f825384222551, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.102645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.124533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 799/1611] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:09 2022] PUT /identity/v3/projects/bd2a81810fcb4a11abc110f5e90052d4/users/06932e0595d442bdaa2f825384222551/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:09.136064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.175621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a5c4ac-f039-4858-b47c-c616192f245d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.178083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a5c4ac-f039-4858-b47c-c616192f245d service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.189596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 813/1612] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:09.191869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.192265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.192600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.220215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:09.221538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:09.238513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:09.243788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88479185-69e4-44f6-a9ce-5af2594ffdd1 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.248414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:09.251873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd0c150-5861-41df-9c2c-fcb9fed89e9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.252548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd0c150-5861-41df-9c2c-fcb9fed89e9d admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.252931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd0c150-5861-41df-9c2c-fcb9fed89e9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.253210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd0c150-5861-41df-9c2c-fcb9fed89e9d admin admin] PATH_INFO: `/v3/projects/594a13a702cf47898a48ab4fd6789359/users/d574235b791e485aa64130c599539945/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.290734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd0c150-5861-41df-9c2c-fcb9fed89e9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=594a13a702cf47898a48ab4fd6789359, user_id=d574235b791e485aa64130c599539945, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.294790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd0c150-5861-41df-9c2c-fcb9fed89e9d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.311881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 800/1613] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:09 2022] PUT /identity/v3/projects/594a13a702cf47898a48ab4fd6789359/users/d574235b791e485aa64130c599539945/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:09.313380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243b041c-b9d7-4593-8a2f-9e6412c954e0 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.313637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243b041c-b9d7-4593-8a2f-9e6412c954e0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.313958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243b041c-b9d7-4593-8a2f-9e6412c954e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.332424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 814/1614] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:09.333615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.338083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:09.341899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-243b041c-b9d7-4593-8a2f-9e6412c954e0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:09.343132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-243b041c-b9d7-4593-8a2f-9e6412c954e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:09.430717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 801/1615] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:09.431169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243b041c-b9d7-4593-8a2f-9e6412c954e0 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.437023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08ad055-daca-4fc9-98c5-46f35385cdfe None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.458947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243b041c-b9d7-4593-8a2f-9e6412c954e0 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.469879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e0869d-ba43-44ef-a6fa-4309401fd6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.470766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0869d-ba43-44ef-a6fa-4309401fd6ed admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.471271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0869d-ba43-44ef-a6fa-4309401fd6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.471834 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e0869d-ba43-44ef-a6fa-4309401fd6ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.472756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e0869d-ba43-44ef-a6fa-4309401fd6ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.474695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e0869d-ba43-44ef-a6fa-4309401fd6ed admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.491691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 802/1616] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:09 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:09.491691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b19dbdf-6d15-40d3-8142-8939c117813d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.491691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b19dbdf-6d15-40d3-8142-8939c117813d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.491691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b19dbdf-6d15-40d3-8142-8939c117813d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.504545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401838a-594c-4226-b245-855f79024d38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.505883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401838a-594c-4226-b245-855f79024d38 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.505883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401838a-594c-4226-b245-855f79024d38 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.505883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401838a-594c-4226-b245-855f79024d38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.518389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401838a-594c-4226-b245-855f79024d38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.520268 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401838a-594c-4226-b245-855f79024d38 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.521841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5b19dbdf-6d15-40d3-8142-8939c117813d None None] MFA Rules not processed for user `06932e0595d442bdaa2f825384222551`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:09.523077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5b19dbdf-6d15-40d3-8142-8939c117813d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:09.532645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 815/1617] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:09.534798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192d66fa-05a7-4865-abb7-98119be974da None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.535321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192d66fa-05a7-4865-abb7-98119be974da None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.535660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192d66fa-05a7-4865-abb7-98119be974da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.556401 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-192d66fa-05a7-4865-abb7-98119be974da None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:09.558434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-192d66fa-05a7-4865-abb7-98119be974da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:09.569728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 816/1618] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:09.569728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192d66fa-05a7-4865-abb7-98119be974da None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.580852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192d66fa-05a7-4865-abb7-98119be974da None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.584874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a89101-a2cf-485f-b8c5-f8fe05ef1589 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.585625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a89101-a2cf-485f-b8c5-f8fe05ef1589 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.586170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a89101-a2cf-485f-b8c5-f8fe05ef1589 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.586170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a89101-a2cf-485f-b8c5-f8fe05ef1589 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.597842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a89101-a2cf-485f-b8c5-f8fe05ef1589 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.599325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a89101-a2cf-485f-b8c5-f8fe05ef1589 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.607908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 803/1619] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:09.609404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b19dbdf-6d15-40d3-8142-8939c117813d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.611163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 817/1620] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:09.625889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511dd85b-6496-4f88-83a3-0d21fa3d93fd None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.626167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b19dbdf-6d15-40d3-8142-8939c117813d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.626307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511dd85b-6496-4f88-83a3-0d21fa3d93fd None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.626503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511dd85b-6496-4f88-83a3-0d21fa3d93fd None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.628017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 818/1621] 10.222.0.33 () {54 vars in 912 bytes} [Thu Oct 6 04:32:09 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:32:09.628733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511dd85b-6496-4f88-83a3-0d21fa3d93fd None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.639870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.640899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.641121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.641497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.653304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511dd85b-6496-4f88-83a3-0d21fa3d93fd None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.657701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.659773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494cf622-8562-4a82-8862-76966db8986e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.660021 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.660496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494cf622-8562-4a82-8862-76966db8986e admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.660836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494cf622-8562-4a82-8862-76966db8986e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.661117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494cf622-8562-4a82-8862-76966db8986e admin admin] PATH_INFO: `/v3/projects/8da5f24e05f04d58a67310fde175410b/users/2ba4589680e74b458ddce087117518ab/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.677685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 804/1622] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:09.678760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.684309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494cf622-8562-4a82-8862-76966db8986e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8da5f24e05f04d58a67310fde175410b, user_id=2ba4589680e74b458ddce087117518ab, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.688629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494cf622-8562-4a82-8862-76966db8986e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.690789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21759b17-0329-4899-b5d7-4673ba2cd604 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.696865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4220f2f-3147-452f-8086-6b57ae45954d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.697563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4220f2f-3147-452f-8086-6b57ae45954d service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.698212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4220f2f-3147-452f-8086-6b57ae45954d service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.698499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4220f2f-3147-452f-8086-6b57ae45954d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.708250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 819/1623] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:09 2022] PUT /identity/v3/projects/8da5f24e05f04d58a67310fde175410b/users/2ba4589680e74b458ddce087117518ab/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:09.709810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b88ae98-5858-4658-b961-10f1cd89dff3 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.710287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b88ae98-5858-4658-b961-10f1cd89dff3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.710709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b88ae98-5858-4658-b961-10f1cd89dff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.711956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4220f2f-3147-452f-8086-6b57ae45954d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.714354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4220f2f-3147-452f-8086-6b57ae45954d service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.729773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 805/1624] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:09.731652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63744cd1-aaa8-4dec-9371-e7881936084e None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.732071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63744cd1-aaa8-4dec-9371-e7881936084e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.732315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63744cd1-aaa8-4dec-9371-e7881936084e None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.734014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 806/1625] 10.222.0.33 () {54 vars in 961 bytes} [Thu Oct 6 04:32:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:32:09.735832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63744cd1-aaa8-4dec-9371-e7881936084e None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.743992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9b88ae98-5858-4658-b961-10f1cd89dff3 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:09.745706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9b88ae98-5858-4658-b961-10f1cd89dff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:09.801222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:09.803925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:09.861215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63744cd1-aaa8-4dec-9371-e7881936084e None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.868510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738d39fc-cbfa-449b-b13d-6cb4d8e00107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.869357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d39fc-cbfa-449b-b13d-6cb4d8e00107 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.870469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d39fc-cbfa-449b-b13d-6cb4d8e00107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.870924 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738d39fc-cbfa-449b-b13d-6cb4d8e00107 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.871663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738d39fc-cbfa-449b-b13d-6cb4d8e00107 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.873818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738d39fc-cbfa-449b-b13d-6cb4d8e00107 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:09.893057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 807/1626] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:09 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:09.894462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f286cb-b2e3-4aa8-8b15-5a0f72cc8f37 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.894752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f286cb-b2e3-4aa8-8b15-5a0f72cc8f37 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.895054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f286cb-b2e3-4aa8-8b15-5a0f72cc8f37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.910093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 820/1627] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:09.911428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b88ae98-5858-4658-b961-10f1cd89dff3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.941339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b88ae98-5858-4658-b961-10f1cd89dff3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:09.942460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-36f286cb-b2e3-4aa8-8b15-5a0f72cc8f37 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:09.944057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-36f286cb-b2e3-4aa8-8b15-5a0f72cc8f37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:09.956516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:09.957166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:09.958348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 808/1628] 10.222.0.33 () {58 vars in 1058 bytes} [Thu Oct 6 04:32:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:09.959427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f286cb-b2e3-4aa8-8b15-5a0f72cc8f37 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:09.967968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:09.968394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] PATH_INFO: `/v3/projects/8da5f24e05f04d58a67310fde175410b/users/2ba4589680e74b458ddce087117518ab/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:09.973129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8da5f24e05f04d58a67310fde175410b, user_id=2ba4589680e74b458ddce087117518ab, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:09.981047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:10.009816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 821/1629] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:09 2022] PUT /identity/v3/projects/8da5f24e05f04d58a67310fde175410b/users/2ba4589680e74b458ddce087117518ab/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:10.011431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:10.048193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f286cb-b2e3-4aa8-8b15-5a0f72cc8f37 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:10.137032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f997357a-5bf1-4b5d-ad62-23943c1a6653 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:10.137672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997357a-5bf1-4b5d-ad62-23943c1a6653 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:10.138327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997357a-5bf1-4b5d-ad62-23943c1a6653 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:10.138605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f997357a-5bf1-4b5d-ad62-23943c1a6653 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:10.181478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a731767-3b63-40ff-8996-6a8503ff9da1 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:10.198597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:10.199282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:10.199558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:10.199828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:10.245040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f997357a-5bf1-4b5d-ad62-23943c1a6653 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:10.247229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f997357a-5bf1-4b5d-ad62-23943c1a6653 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:10.265915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 809/1630] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:10.267484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1454c-bdab-4029-b2a5-ad64035b9030 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:10.268470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1454c-bdab-4029-b2a5-ad64035b9030 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:10.268898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1454c-bdab-4029-b2a5-ad64035b9030 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:10.318683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d5d1454c-bdab-4029-b2a5-ad64035b9030 None None] MFA Rules not processed for user `2ba4589680e74b458ddce087117518ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:10.320691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d5d1454c-bdab-4029-b2a5-ad64035b9030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:10.324513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:10.326122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:10.332867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:10.359375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 822/1631] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:10 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:10.462103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 810/1632] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:10 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:10.489031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:10.504583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b394f9f6-b333-40cd-99c0-7241647db212 service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:10.516237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5e7593-71c8-465d-b8e0-bb17e6bc4a29 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:10.516863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5e7593-71c8-465d-b8e0-bb17e6bc4a29 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:10.517138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5e7593-71c8-465d-b8e0-bb17e6bc4a29 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:10.517414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5e7593-71c8-465d-b8e0-bb17e6bc4a29 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:10.538072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5e7593-71c8-465d-b8e0-bb17e6bc4a29 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:10.539549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5e7593-71c8-465d-b8e0-bb17e6bc4a29 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:10.569497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 823/1633] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:10 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:10.735148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d1454c-bdab-4029-b2a5-ad64035b9030 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:10.760075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d1454c-bdab-4029-b2a5-ad64035b9030 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:10.793749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:10.794764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:10.795319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:10.796567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:10.862700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b9fc58-4ac1-4cae-88e0-9756a76416ee None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:10.863900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b9fc58-4ac1-4cae-88e0-9756a76416ee None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:10.864549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b9fc58-4ac1-4cae-88e0-9756a76416ee None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:10.867435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 824/1634] 10.222.0.33 () {54 vars in 912 bytes} [Thu Oct 6 04:32:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:32:10.885854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d68acb-5a97-4714-ae01-b87f6666ddb8 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:10.886766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d68acb-5a97-4714-ae01-b87f6666ddb8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:10.887265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d68acb-5a97-4714-ae01-b87f6666ddb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:10.934485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:10.934879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f1d68acb-5a97-4714-ae01-b87f6666ddb8 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:10.936989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:10.937957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f1d68acb-5a97-4714-ae01-b87f6666ddb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:10.951733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 811/1635] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:10 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:10.975829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 825/1636] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:32:10 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:11.228028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:11.254847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d0d39-7716-43fe-9d3b-3da9052dffcf service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:11.264712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a492878-038f-4f7a-a1f4-b4bb4f83ffd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:11.265633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a492878-038f-4f7a-a1f4-b4bb4f83ffd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:11.268374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a492878-038f-4f7a-a1f4-b4bb4f83ffd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:11.270306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a492878-038f-4f7a-a1f4-b4bb4f83ffd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:11.296135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a492878-038f-4f7a-a1f4-b4bb4f83ffd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:11.300057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a492878-038f-4f7a-a1f4-b4bb4f83ffd3 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:11.329145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 812/1637] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:11.662107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d68acb-5a97-4714-ae01-b87f6666ddb8 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:11.732930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:11.817932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d68acb-5a97-4714-ae01-b87f6666ddb8 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:11.835840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:11.837838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:11.838164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:11.841153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] PATH_INFO: `/v3/users/06932e0595d442bdaa2f825384222551` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:11.842635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06932e0595d442bdaa2f825384222551)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:11.844628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:11.956765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06932e0595d442bdaa2f825384222551'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:11.958243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06932e0595d442bdaa2f825384222551'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:11.984523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06932e0595d442bdaa2f825384222551'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:12.015771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06932e0595d442bdaa2f825384222551'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:12.017843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06932e0595d442bdaa2f825384222551'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:12.044104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:12.044998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:12.045494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:12.050885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 826/1638] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:11 2022] DELETE /identity/v3/users/06932e0595d442bdaa2f825384222551 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:12.072660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:12.074270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:12.198200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 813/1639] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:12.248562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:12.250779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:12.407999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d5e809-c53d-4ccf-b3d0-04351136ab02 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:12.423716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52def71-5fb7-47ba-8362-a6a2e1ef1501 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:12.424576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52def71-5fb7-47ba-8362-a6a2e1ef1501 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:12.424904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52def71-5fb7-47ba-8362-a6a2e1ef1501 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:12.425266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52def71-5fb7-47ba-8362-a6a2e1ef1501 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:12.444853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52def71-5fb7-47ba-8362-a6a2e1ef1501 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:12.449041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52def71-5fb7-47ba-8362-a6a2e1ef1501 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:12.465792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 827/1640] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:14.775178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:14.777412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:14.802819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfd2845-6fda-401b-91dc-c7d9da7020c0 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:14.812069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:14.812994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:14.813483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:14.813910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] PATH_INFO: `/v3/projects/bd2a81810fcb4a11abc110f5e90052d4` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:14.815511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd2a81810fcb4a11abc110f5e90052d4)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:14.830730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:14.907292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:14.940942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd2a81810fcb4a11abc110f5e90052d4', 'request_id': 'req-5d7f380f-6a2e-4809-9f19-e7815398e563'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:14.954874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd2a81810fcb4a11abc110f5e90052d4', 'request_id': 'req-5d7f380f-6a2e-4809-9f19-e7815398e563'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:14.965783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd2a81810fcb4a11abc110f5e90052d4', 'request_id': 'req-5d7f380f-6a2e-4809-9f19-e7815398e563'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.003075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] The token cache is being invalidate because project bd2a81810fcb4a11abc110f5e90052d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:15.003600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.004448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5d7f380f-6a2e-4809-9f19-e7815398e563 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.010225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 814/1641] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:14 2022] DELETE /identity/v3/projects/bd2a81810fcb4a11abc110f5e90052d4 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:15.011507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb92955-d25d-4ac4-a0a5-f6d11e035dff None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.011971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb92955-d25d-4ac4-a0a5-f6d11e035dff None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.012154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb92955-d25d-4ac4-a0a5-f6d11e035dff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.060554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-feb92955-d25d-4ac4-a0a5-f6d11e035dff None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:15.061966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-feb92955-d25d-4ac4-a0a5-f6d11e035dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:15.157562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:15.184002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 828/1642] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:15.191040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2836988-e85a-4612-a4a5-8c5b2bcd11e6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.191333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2836988-e85a-4612-a4a5-8c5b2bcd11e6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.191656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2836988-e85a-4612-a4a5-8c5b2bcd11e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.244061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a2836988-e85a-4612-a4a5-8c5b2bcd11e6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:15.249879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a2836988-e85a-4612-a4a5-8c5b2bcd11e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:15.285459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 815/1643] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:15.285801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb92955-d25d-4ac4-a0a5-f6d11e035dff None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:15.304839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb92955-d25d-4ac4-a0a5-f6d11e035dff None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:15.315843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:15.316896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.317187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.317462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.318133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:15.318434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:15.320351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:15.339456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 829/1644] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:15.346031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2836988-e85a-4612-a4a5-8c5b2bcd11e6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:15.359364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2836988-e85a-4612-a4a5-8c5b2bcd11e6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:15.373429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:15.374067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.374344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.374613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.375308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:15.383299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:15.430651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:15.433317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:15.463964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 816/1645] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:15 2022] POST /identity/v3/projects => generated 405 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:15.464574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db087750-7416-4024-8a4a-22c1f599fba0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:15.465943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:15.669477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8d25fd-db70-4e0d-8442-4f092b3d6e8f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:15.670194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:15.671758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.672044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.673243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] PATH_INFO: `/v3/users/2c6627dc1a8c4a7197c3a1d3f75decc5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.679945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:15.680581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.680864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.681091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c6627dc1a8c4a7197c3a1d3f75decc5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:15.681188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.681772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:15.687137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:15.688680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:15.734497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 817/1646] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:15 2022] POST /identity/v3/users => generated 303 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:15.738632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:15.765943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac6271-30c1-42c2-a012-711caee5c557 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:15.781703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:15.783190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.783320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c6627dc1a8c4a7197c3a1d3f75decc5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.784090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.784201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c6627dc1a8c4a7197c3a1d3f75decc5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.787044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.795513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c6627dc1a8c4a7197c3a1d3f75decc5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.796147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:15.797960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:15.826460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c6627dc1a8c4a7197c3a1d3f75decc5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.827328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3fe85de0-07ac-4863-a296-262d4b28107e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c6627dc1a8c4a7197c3a1d3f75decc5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:15.828427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 818/1647] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:15 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:32:15.835212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:15.850101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:15.858777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 830/1648] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:15 2022] DELETE /identity/v3/users/2c6627dc1a8c4a7197c3a1d3f75decc5 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:32:15.863414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:15.863931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:15.864689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:15.885489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c43b9e-61fe-4fc6-9ca4-a59e582314af admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:15.943799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:15.946880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:16.042487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:16.043146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:16.043430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:16.043874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:16.044532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:16.049016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:16.066076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=6,keystone:SELECT=89,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:16.084043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 819/1649] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:15 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:16.084978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:16.108880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5a11e4-29f3-4f3e-be67-b19662cfeaba admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:16.116452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 831/1650] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:16.124618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271f4fef-a876-4c6d-81a3-da8c683fa083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:16.127264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f4fef-a876-4c6d-81a3-da8c683fa083 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:16.127552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f4fef-a876-4c6d-81a3-da8c683fa083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:16.127832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f4fef-a876-4c6d-81a3-da8c683fa083 admin admin] PATH_INFO: `/v3/projects/56d175f104054e7d882def198101076d/users/b1f54f0d4e8b4f839db11cc7985b5562/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:16.144353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:16.145982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:16.153502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:16.192692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271f4fef-a876-4c6d-81a3-da8c683fa083 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56d175f104054e7d882def198101076d, user_id=b1f54f0d4e8b4f839db11cc7985b5562, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:16.202583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271f4fef-a876-4c6d-81a3-da8c683fa083 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:16.238312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 820/1651] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:16 2022] PUT /identity/v3/projects/56d175f104054e7d882def198101076d/users/b1f54f0d4e8b4f839db11cc7985b5562/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:16.253089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77aafacc-5e57-4eff-baf3-4546c88176e8 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:16.254330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77aafacc-5e57-4eff-baf3-4546c88176e8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:16.254762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77aafacc-5e57-4eff-baf3-4546c88176e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:16.297735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011e0023-1bbe-44cd-be27-4fabcfc7e308 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:16.317913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-77aafacc-5e57-4eff-baf3-4546c88176e8 None None] MFA Rules not processed for user `b1f54f0d4e8b4f839db11cc7985b5562`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:16.324179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-77aafacc-5e57-4eff-baf3-4546c88176e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:16.403814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:16.404096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:16.404224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:16.404491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:16.474688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 821/1652] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:16 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:16.479551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77aafacc-5e57-4eff-baf3-4546c88176e8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:16.539484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:16.542180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:16.567010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 832/1653] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:32:16.604305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77aafacc-5e57-4eff-baf3-4546c88176e8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:16.610881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:16.612113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:16.612563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:16.612963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:16.613744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:16.616612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:16.668211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 822/1654] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:16 2022] POST /identity/v3/projects => generated 405 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:16.672369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:16.690117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36404720-9a80-4df3-9f59-59b510770495 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:16.850747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:16.852026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:16.852470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:16.853008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:16.854410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:16.858520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:16.887824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:16.897655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 833/1655] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:16 2022] POST /identity/v3/users => generated 309 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:16.902199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:16.924004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7715c-2a67-4ccf-8e88-fafb831f3cee admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:16.935943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:16.936579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:16.938479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:16.940219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:16.940929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:16.945765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:16.983240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 823/1656] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:16 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:16.984837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:17.001659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586c2f66-c819-43b5-b46d-d1efa4cb2de5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:17.019158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:17.021334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:17.021626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:17.023150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] PATH_INFO: `/v3/projects/0403021eb9b54bd8b4f4bdd7007dba45/users/12a9716644e945f3bd4fdb6e1ca3f479/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:17.052526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0403021eb9b54bd8b4f4bdd7007dba45, user_id=12a9716644e945f3bd4fdb6e1ca3f479, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:17.056207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:17.105993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 834/1657] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:16 2022] PUT /identity/v3/projects/0403021eb9b54bd8b4f4bdd7007dba45/users/12a9716644e945f3bd4fdb6e1ca3f479/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:17.117768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:17.162198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573bd0c8-3799-4a2f-bcef-9e2328ac125d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:17.257464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:17.331121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025d4302-7939-40bd-8a2c-84743fe5c22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:17.332144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d4302-7939-40bd-8a2c-84743fe5c22d admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:17.332809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d4302-7939-40bd-8a2c-84743fe5c22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:17.333899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d4302-7939-40bd-8a2c-84743fe5c22d admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:17.335196 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025d4302-7939-40bd-8a2c-84743fe5c22d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:17.337095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025d4302-7939-40bd-8a2c-84743fe5c22d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:17.358648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 824/1658] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:17 2022] GET /identity/v3/roles => generated 1420 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:17.359045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:17.371364 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee8b3f-1c01-410b-90d4-baf54f536ce9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:17.380026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:17.381704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:17.381704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:17.381704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] PATH_INFO: `/v3/projects/0403021eb9b54bd8b4f4bdd7007dba45/users/12a9716644e945f3bd4fdb6e1ca3f479/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:17.383648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0403021eb9b54bd8b4f4bdd7007dba45, user_id=12a9716644e945f3bd4fdb6e1ca3f479, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:17.388233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:17.403009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 835/1659] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:17 2022] PUT /identity/v3/projects/0403021eb9b54bd8b4f4bdd7007dba45/users/12a9716644e945f3bd4fdb6e1ca3f479/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:17.416825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:17.417282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:17.417665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:17.465459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] MFA Rules not processed for user `12a9716644e945f3bd4fdb6e1ca3f479`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:17.466905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:17.618677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 825/1660] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:17 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:17.626084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:17.627950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:17.764333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915a572-d3b7-4c5c-a04f-7230af5e5862 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:17.783705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f543cf-6bd7-4b73-af30-82d060c342a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:17.784417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f543cf-6bd7-4b73-af30-82d060c342a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:17.784623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f543cf-6bd7-4b73-af30-82d060c342a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:17.784866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f543cf-6bd7-4b73-af30-82d060c342a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:17.826227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f543cf-6bd7-4b73-af30-82d060c342a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:17.828002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f543cf-6bd7-4b73-af30-82d060c342a8 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:17.852573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 836/1661] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:17 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:19.407624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:19.410278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:19.660936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd4d3bb-4ee3-459f-ba32-9d436e60bde1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:19.675684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:19.676355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:19.676637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:19.676900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] PATH_INFO: `/v3/projects/0b73eb6e32754817806e3b97b0d1b402` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:19.680254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b73eb6e32754817806e3b97b0d1b402)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:19.683950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:19.795714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b73eb6e32754817806e3b97b0d1b402', 'request_id': 'req-c8287103-e02a-4abe-ad72-557119b24292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:19.814789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b73eb6e32754817806e3b97b0d1b402', 'request_id': 'req-c8287103-e02a-4abe-ad72-557119b24292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:19.844625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b73eb6e32754817806e3b97b0d1b402', 'request_id': 'req-c8287103-e02a-4abe-ad72-557119b24292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:19.846920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:19.888817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] The token cache is being invalidate because project 0b73eb6e32754817806e3b97b0d1b402 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:19.889896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:19.891367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c8287103-e02a-4abe-ad72-557119b24292 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:19.898194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 826/1662] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:19 2022] DELETE /identity/v3/projects/0b73eb6e32754817806e3b97b0d1b402 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:19.900985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f228a014-e77e-4f52-87af-eeaf9f8c7c1a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:19.901516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f228a014-e77e-4f52-87af-eeaf9f8c7c1a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:19.901662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f228a014-e77e-4f52-87af-eeaf9f8c7c1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:19.966906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f228a014-e77e-4f52-87af-eeaf9f8c7c1a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:19.972140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f228a014-e77e-4f52-87af-eeaf9f8c7c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:20.202235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 837/1663] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:20.208375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebe84c-36d5-4ebd-8795-8e5c813d4870 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:20.208968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebe84c-36d5-4ebd-8795-8e5c813d4870 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:20.209461 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebe84c-36d5-4ebd-8795-8e5c813d4870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:20.264248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c0ebe84c-36d5-4ebd-8795-8e5c813d4870 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:20.265548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c0ebe84c-36d5-4ebd-8795-8e5c813d4870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:20.278127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 827/1664] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:20.282654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f228a014-e77e-4f52-87af-eeaf9f8c7c1a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:20.295713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f228a014-e77e-4f52-87af-eeaf9f8c7c1a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:20.307975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:20.308737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:20.309113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:20.310126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:20.311270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:20.311782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:20.313991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:20.339770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ebe84c-36d5-4ebd-8795-8e5c813d4870 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:20.347663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 838/1665] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:20.352355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ebe84c-36d5-4ebd-8795-8e5c813d4870 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:20.370065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:20.371729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:20.372019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:20.372768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:20.373486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:20.376130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:20.440111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 828/1666] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:20 2022] POST /identity/v3/projects => generated 367 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:20.442075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:20.453661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5fa99d-2b55-4909-b388-580d1b58975c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:20.618570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:20.653228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:20.653784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:20.654081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:20.654341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:20.658881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:20.660930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:20.727825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 839/1667] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:20 2022] POST /identity/v3/users => generated 284 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:20.731528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:20.750430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a62030-c8b0-4c97-986a-3015835d26c1 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:20.764788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:20.766097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:20.768902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:20.769265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:20.769978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:20.771795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:20.803936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 829/1668] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:20 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:32:20.808452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:20.822261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3665f-c9ae-4442-8208-a574f36f7c32 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:20.849344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5198e4f0-553c-40f5-8243-c718c0e359cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:20.849981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5198e4f0-553c-40f5-8243-c718c0e359cc admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:20.850744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5198e4f0-553c-40f5-8243-c718c0e359cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:20.851058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5198e4f0-553c-40f5-8243-c718c0e359cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:20.851730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5198e4f0-553c-40f5-8243-c718c0e359cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:20.854295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5198e4f0-553c-40f5-8243-c718c0e359cc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:20.896137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 840/1669] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:20 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:20.899431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:20.918309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec70fd9-67ce-42e3-bb2a-271554864653 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:20.938160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:20.942778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:20.943091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:20.943373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] PATH_INFO: `/v3/projects/38c90b6829624e059b5ae1afed5a34b9/users/031e7a50ee4d4ff48d8ad593a924db4e/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:20.985963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38c90b6829624e059b5ae1afed5a34b9, user_id=031e7a50ee4d4ff48d8ad593a924db4e, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:20.989863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:21.027270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 830/1670] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:20 2022] PUT /identity/v3/projects/38c90b6829624e059b5ae1afed5a34b9/users/031e7a50ee4d4ff48d8ad593a924db4e/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:21.038989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe345c-0ed3-4345-ae7c-21a9bf3948f7 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:21.039481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe345c-0ed3-4345-ae7c-21a9bf3948f7 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:21.039614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe345c-0ed3-4345-ae7c-21a9bf3948f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:21.127585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7bfe345c-0ed3-4345-ae7c-21a9bf3948f7 None None] MFA Rules not processed for user `031e7a50ee4d4ff48d8ad593a924db4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:21.128980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7bfe345c-0ed3-4345-ae7c-21a9bf3948f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:21.361524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 841/1671] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:21.361907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:21.503607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e19a73e-1b04-4dd9-8c76-ec873c563475 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:21.515603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:21.516184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:21.516532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:21.516872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:21.517618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:21.519922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:21.581836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfe345c-0ed3-4345-ae7c-21a9bf3948f7 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:21.584705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 831/1672] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:21 2022] POST /identity/v3/projects => generated 369 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:21.602312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfe345c-0ed3-4345-ae7c-21a9bf3948f7 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:21.720179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:21.721351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:21.721810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:21.722247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:21.723677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:21.728314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:21.795160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 842/1673] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:21 2022] POST /identity/v3/users => generated 291 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:21.799276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:21.820670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:21.824804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:21.832724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ff12ee-0c30-4cc1-8f78-e4f44a3bb7ac admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:21.873819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:21.874895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:21.875435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:21.875910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:21.876946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:21.877519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bdb616-4a08-42f3-8f34-707d4e61c83e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:21.883060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:21.891170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:21.891831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:21.893308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:21.893611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] PATH_INFO: `/v3/users/b1f54f0d4e8b4f839db11cc7985b5562` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:21.897761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1f54f0d4e8b4f839db11cc7985b5562)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:21.903364 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:21.932494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 832/1674] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:21 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:21.936422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:21.951426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8298bfa-4dfa-43bb-a4e1-3415d783a64c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:21.962854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:21.972935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:21.974057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:21.976204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] PATH_INFO: `/v3/projects/18db6d5ee5c44bf8bdbd4208c1d7cfca/users/43789128dcdf4e838929595a875bf0fe/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:22.021143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] RBAC: Authorizing `identity:create_grant(project_id=18db6d5ee5c44bf8bdbd4208c1d7cfca, user_id=43789128dcdf4e838929595a875bf0fe, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:22.029165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=88,keystone:INSERT=6,keystone:DELETE=6 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:22.031701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:22.037954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1f54f0d4e8b4f839db11cc7985b5562'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:22.039963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1f54f0d4e8b4f839db11cc7985b5562'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:22.068646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1f54f0d4e8b4f839db11cc7985b5562'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:22.069315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 833/1675] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:21 2022] PUT /identity/v3/projects/18db6d5ee5c44bf8bdbd4208c1d7cfca/users/43789128dcdf4e838929595a875bf0fe/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:22.074570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:22.111882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1f54f0d4e8b4f839db11cc7985b5562'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:22.115342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d769a700-9b75-4fd1-ae11-22be2def2ca0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1f54f0d4e8b4f839db11cc7985b5562'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:22.152540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 843/1676] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:21 2022] DELETE /identity/v3/users/b1f54f0d4e8b4f839db11cc7985b5562 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:22.154924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:22.155460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:22.157029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:22.191376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:INSERT=5,keystone:DELETE=1,keystone:SELECT=94 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:22.217790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:22.221834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:22.250349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542bf00d-a3a5-4bcd-a44d-6c3f42133afc admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:22.391711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 844/1677] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:22.399800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:22.400416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:22.401433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:22.401790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:22.402453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:22.404102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:22.428050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 834/1678] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:22 2022] GET /identity/v3/roles => generated 1420 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:22.430997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:22.433427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:22.439265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:22.441052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:22.458304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c0d9ac-e07c-4412-9d50-f0127a92326b None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:22.469949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729f1b8-b2c6-40ca-ad45-f1e1a932e129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:22.470628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729f1b8-b2c6-40ca-ad45-f1e1a932e129 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:22.470904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729f1b8-b2c6-40ca-ad45-f1e1a932e129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:22.471199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729f1b8-b2c6-40ca-ad45-f1e1a932e129 admin admin] PATH_INFO: `/v3/projects/18db6d5ee5c44bf8bdbd4208c1d7cfca/users/43789128dcdf4e838929595a875bf0fe/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:22.476661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729f1b8-b2c6-40ca-ad45-f1e1a932e129 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18db6d5ee5c44bf8bdbd4208c1d7cfca, user_id=43789128dcdf4e838929595a875bf0fe, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:22.479877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729f1b8-b2c6-40ca-ad45-f1e1a932e129 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:22.507091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 845/1679] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:22 2022] PUT /identity/v3/projects/18db6d5ee5c44bf8bdbd4208c1d7cfca/users/43789128dcdf4e838929595a875bf0fe/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:22.519309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:22.519737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:22.520023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:22.572131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10837a0b-2dfa-4a79-8140-90bd760e3d6d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:22.591150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] MFA Rules not processed for user `43789128dcdf4e838929595a875bf0fe`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:22.592419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:22.693598 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:22.694573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:22.694856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:22.695150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:22.766448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 846/1680] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:22 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:22.784408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:22.786677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:22.792472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:22.823630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:22.825743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:22.850168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 835/1681] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:32:22.878800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:22.951626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a70cb27-be62-453b-a03b-b7c09e8814b3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:22.961192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41330c83-392f-4c5b-9584-ceb5a653017e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:22.963714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41330c83-392f-4c5b-9584-ceb5a653017e service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:22.965710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41330c83-392f-4c5b-9584-ceb5a653017e service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:22.966128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41330c83-392f-4c5b-9584-ceb5a653017e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:23.009412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41330c83-392f-4c5b-9584-ceb5a653017e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:23.012944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41330c83-392f-4c5b-9584-ceb5a653017e service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:23.038746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 847/1682] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:22 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:23.160352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:23.163566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:23.193242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741687a-3076-478d-b2d7-40f47040e33a service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:23.205704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:23.208207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:23.219820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:23.222544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:23.234257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:23.430227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:23.437673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:23.467237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 836/1683] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:23 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:24.321764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c708cbf4-848a-42ea-b335-c28e84c6888b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:24.322779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c708cbf4-848a-42ea-b335-c28e84c6888b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:24.323166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c708cbf4-848a-42ea-b335-c28e84c6888b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:24.420430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c708cbf4-848a-42ea-b335-c28e84c6888b None None] MFA Rules not processed for user `56c86f73f98a4fb2af787534fd47e2ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:24.422009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c708cbf4-848a-42ea-b335-c28e84c6888b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:24.602126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 848/1684] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:24.622375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:24.625056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:24.873824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cefa13-23d0-4967-90e9-c9d6078c12e0 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:24.891294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:24.891926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:24.892192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:24.892462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:24.933510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:24.935508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:24.950008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 837/1685] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:24 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:25.438734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:25.439946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:25.440379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:25.488061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:25.489553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:25.522921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de439748-80b5-4ee3-86ec-c2857bca9d57 service nova] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:25.541041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:25.545275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:25.545275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:25.545275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] PATH_INFO: `/v3/projects/56d175f104054e7d882def198101076d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:25.545275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] RBAC: Authorizing `identity:delete_project(project_id=56d175f104054e7d882def198101076d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:25.563240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:25.570468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] MFA Rules not processed for user `d574235b791e485aa64130c599539945`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:25.573551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:25.725053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56d175f104054e7d882def198101076d', 'request_id': 'req-02c54079-18ce-46bd-9801-41739a6532de'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:25.740750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56d175f104054e7d882def198101076d', 'request_id': 'req-02c54079-18ce-46bd-9801-41739a6532de'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:25.768114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56d175f104054e7d882def198101076d', 'request_id': 'req-02c54079-18ce-46bd-9801-41739a6532de'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:25.773395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 849/1686] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:25 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:25.776461 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:25.796236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:25.797951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:25.800922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] The token cache is being invalidate because project 56d175f104054e7d882def198101076d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:25.803504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:25.809941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:25.813655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 838/1687] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:25 2022] DELETE /identity/v3/projects/56d175f104054e7d882def198101076d => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:32:25.819011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:25.823925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:25.850785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f54faa-4a77-4e68-a212-34e6d11e15dc None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:26.077785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:26.095574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c54079-18ce-46bd-9801-41739a6532de admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:26.109978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:26.111219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:26.113232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:26.113644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] PATH_INFO: `/v3/users/12a9716644e945f3bd4fdb6e1ca3f479` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:26.118868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12a9716644e945f3bd4fdb6e1ca3f479)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:26.122681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: WARNING py.warnings [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:32:26.122681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: warnings.warn(msg) Oct 06 04:32:26.122681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: Oct 06 04:32:26.123894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:26.147193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d78cb0-a931-4099-b459-7e49ad61749e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:26.148031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d78cb0-a931-4099-b459-7e49ad61749e service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:26.148188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d78cb0-a931-4099-b459-7e49ad61749e service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:26.148445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d78cb0-a931-4099-b459-7e49ad61749e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:26.183040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-50d78cb0-a931-4099-b459-7e49ad61749e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:26.382695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12a9716644e945f3bd4fdb6e1ca3f479'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:26.389770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12a9716644e945f3bd4fdb6e1ca3f479'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:26.437860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12a9716644e945f3bd4fdb6e1ca3f479'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:26.439271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12a9716644e945f3bd4fdb6e1ca3f479'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:26.456595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d78cb0-a931-4099-b459-7e49ad61749e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:26.473377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d78cb0-a931-4099-b459-7e49ad61749e service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:26.506117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e1c06bea-4cbf-4094-91ea-c91b34445129 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12a9716644e945f3bd4fdb6e1ca3f479'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:26.523715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 850/1688] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:25 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 727 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:26.548593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:26.551123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:26.553341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:26.555620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 839/1689] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:25 2022] DELETE /identity/v3/users/12a9716644e945f3bd4fdb6e1ca3f479 => generated 0 bytes in 734 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:26.557651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406ce6f2-579d-4532-acba-1fce877d4635 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:26.557937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406ce6f2-579d-4532-acba-1fce877d4635 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:26.558209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406ce6f2-579d-4532-acba-1fce877d4635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:26.640606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-406ce6f2-579d-4532-acba-1fce877d4635 None None] MFA Rules not processed for user `4f29428b32454dbdb9c644759389abec`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:26.641853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-406ce6f2-579d-4532-acba-1fce877d4635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:26.697151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] MFA Rules not processed for user `f68d73b5efc74b3380429cf72e506905`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:26.702849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:26.893894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 840/1690] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:26.905271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:26.905903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:26.906533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:26.920369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:26.951572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] MFA Rules not processed for user `4f29428b32454dbdb9c644759389abec`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:26.956153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:26.974621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 841/1691] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:26.990303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:26.991848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:27.081064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 851/1692] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 537 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:27.087757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:27.089982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:27.304973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b4fc06-749f-4578-87ff-3594e3942552 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:27.308404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae1fb44-9297-449d-93b5-c751dbf85e7c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:27.319433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:27.330666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263a3c72-4f7f-438e-8074-5dc398d44d0f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:27.331335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a3c72-4f7f-438e-8074-5dc398d44d0f service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:27.331612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a3c72-4f7f-438e-8074-5dc398d44d0f service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:27.331885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a3c72-4f7f-438e-8074-5dc398d44d0f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:27.340602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:27.341632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:27.341916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:27.351201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263a3c72-4f7f-438e-8074-5dc398d44d0f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:27.352836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263a3c72-4f7f-438e-8074-5dc398d44d0f service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:27.354439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:27.357787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:27.375130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 852/1693] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:32:27.402093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 842/1694] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:26 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:27.466342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:27.474881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:27.511368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:27.709275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab33888-972a-42e0-8519-4656bd40e2be service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:27.725070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:27.726214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:27.726495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:27.726774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] PATH_INFO: `/v3/projects/0403021eb9b54bd8b4f4bdd7007dba45` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:27.727967 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] RBAC: Authorizing `identity:delete_project(project_id=0403021eb9b54bd8b4f4bdd7007dba45)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:27.731969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: WARNING py.warnings [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 06 04:32:27.731969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: warnings.warn(msg) Oct 06 04:32:27.731969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: Oct 06 04:32:27.733193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:27.899277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0403021eb9b54bd8b4f4bdd7007dba45', 'request_id': 'req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:27.939948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0403021eb9b54bd8b4f4bdd7007dba45', 'request_id': 'req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:27.956872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0403021eb9b54bd8b4f4bdd7007dba45', 'request_id': 'req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:27.983501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] The token cache is being invalidate because project 0403021eb9b54bd8b4f4bdd7007dba45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:27.983801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:27.986419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:27.992563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 853/1695] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:27 2022] DELETE /identity/v3/projects/0403021eb9b54bd8b4f4bdd7007dba45 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:27.995100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6dfe22-9cf9-4779-8d4c-be0671dd210b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:27.995988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6dfe22-9cf9-4779-8d4c-be0671dd210b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:27.996520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6dfe22-9cf9-4779-8d4c-be0671dd210b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.050886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-8c6dfe22-9cf9-4779-8d4c-be0671dd210b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:28.050886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8c6dfe22-9cf9-4779-8d4c-be0671dd210b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:28.139962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:28.141737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:28.328730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 843/1696] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:28.330214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c0e67f-eaf7-4b5e-877d-b9805167aaeb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:28.336479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e4c8c-84ed-4e32-b8d5-58a2b1c657a1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:28.336820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e4c8c-84ed-4e32-b8d5-58a2b1c657a1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:28.337080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e4c8c-84ed-4e32-b8d5-58a2b1c657a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.355977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:28.362150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:28.362680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:28.364284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] PATH_INFO: `/v3/users/1847fad78ead4f41b5705b2c555dacee` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.369833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1847fad78ead4f41b5705b2c555dacee)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:28.376178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:28.396090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ae1e4c8c-84ed-4e32-b8d5-58a2b1c657a1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:28.397426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ae1e4c8c-84ed-4e32-b8d5-58a2b1c657a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:28.422433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 844/1697] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:28.427121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e4c8c-84ed-4e32-b8d5-58a2b1c657a1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:28.447301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e4c8c-84ed-4e32-b8d5-58a2b1c657a1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:28.463400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:28.464472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:28.465030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:28.466563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.467712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:28.468208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:28.470262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:28.490538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 845/1698] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:28.507946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:28.523368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b2cba1-d24c-47a0-9f4b-82454492b436 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:28.529679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:28.548102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:28.549161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:28.550386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:28.550812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.551797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:28.553915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1847fad78ead4f41b5705b2c555dacee'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:28.555329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:28.560200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1847fad78ead4f41b5705b2c555dacee'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:28.591373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1847fad78ead4f41b5705b2c555dacee'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:28.606565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 846/1699] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:28 2022] POST /identity/v3/projects => generated 363 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:28.611307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:28.614276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1847fad78ead4f41b5705b2c555dacee'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:28.615271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-632aa703-3b5a-4050-ba3e-50e06e19b5b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1847fad78ead4f41b5705b2c555dacee'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:28.640707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 854/1700] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:28 2022] DELETE /identity/v3/users/1847fad78ead4f41b5705b2c555dacee => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:28.645869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9110cbc-342f-40c0-8a2d-67135ceb1d79 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:28.646446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9110cbc-342f-40c0-8a2d-67135ceb1d79 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:28.646887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9110cbc-342f-40c0-8a2d-67135ceb1d79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.676210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d9110cbc-342f-40c0-8a2d-67135ceb1d79 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:28.677847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d9110cbc-342f-40c0-8a2d-67135ceb1d79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:28.728667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:28.797027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0d684-afd8-4bbc-8952-f187adbe4071 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:28.948868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 855/1701] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:28.951637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:28.953685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:28.953966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.963908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7862438b-1c86-4404-b311-6b0d5ac94e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:28.963908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7862438b-1c86-4404-b311-6b0d5ac94e8c admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:28.964351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7862438b-1c86-4404-b311-6b0d5ac94e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:28.964730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7862438b-1c86-4404-b311-6b0d5ac94e8c admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:28.965538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7862438b-1c86-4404-b311-6b0d5ac94e8c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:28.969970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7862438b-1c86-4404-b311-6b0d5ac94e8c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:29.038994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 847/1702] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:28 2022] POST /identity/v3/users => generated 282 bytes in 426 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:29.042232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] MFA Rules not processed for user `a58cd041c02d46dcb8acf6adb11ee9de`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:29.042799 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf708af-76e0-4211-abf3-38f6e9483233 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.043316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf708af-76e0-4211-abf3-38f6e9483233 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.043923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf708af-76e0-4211-abf3-38f6e9483233 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.045454 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 848/1703] 10.222.0.33 () {54 vars in 912 bytes} [Thu Oct 6 04:32:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:32:29.046481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf708af-76e0-4211-abf3-38f6e9483233 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:29.047018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.048986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5bf708af-76e0-4211-abf3-38f6e9483233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.199324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf708af-76e0-4211-abf3-38f6e9483233 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:29.209568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9295ba-af43-4c18-a863-12db7fed2f9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:29.210627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9295ba-af43-4c18-a863-12db7fed2f9d service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.210923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9295ba-af43-4c18-a863-12db7fed2f9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.212406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9295ba-af43-4c18-a863-12db7fed2f9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.240706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9295ba-af43-4c18-a863-12db7fed2f9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:29.241892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9295ba-af43-4c18-a863-12db7fed2f9d service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:29.264762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 856/1704] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:28 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:29.264762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:29.265939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.266506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 849/1705] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:29.271451 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae6640-4595-42b2-98d6-9eb52deb9133 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.271451 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae6640-4595-42b2-98d6-9eb52deb9133 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.272425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdae6640-4595-42b2-98d6-9eb52deb9133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.295776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a88574-d8a4-42e2-af2f-37a9df9e3c7d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:29.310815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:29.311431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.311710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.311981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.312677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:29.319697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:29.360893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 857/1706] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:29 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:29.362006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:29.365591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.387498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2980e736-8ee6-40e2-9b34-3bdb6405a8a6 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:29.393886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:29.394808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.395249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.395771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] PATH_INFO: `/v3/users/031e7a50ee4d4ff48d8ad593a924db4e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.397404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] RBAC: Authorizing `identity:delete_user(user_id=031e7a50ee4d4ff48d8ad593a924db4e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:29.402476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:29.412559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-cdae6640-4595-42b2-98d6-9eb52deb9133 None None] MFA Rules not processed for user `87829678318948f3b81bd107076bdfa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:29.413789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cdae6640-4595-42b2-98d6-9eb52deb9133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.433943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 850/1707] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:32:29 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:29.435336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d812a-8aa9-4bdd-b82d-095eec07d21b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.438235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d812a-8aa9-4bdd-b82d-095eec07d21b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.438579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d812a-8aa9-4bdd-b82d-095eec07d21b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.477976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:29.491949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-854d812a-8aa9-4bdd-b82d-095eec07d21b None None] MFA Rules not processed for user `a58cd041c02d46dcb8acf6adb11ee9de`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:29.493954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-854d812a-8aa9-4bdd-b82d-095eec07d21b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.500120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '031e7a50ee4d4ff48d8ad593a924db4e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:29.501359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '031e7a50ee4d4ff48d8ad593a924db4e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:29.511881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 851/1708] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:29 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:29.513124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854d812a-8aa9-4bdd-b82d-095eec07d21b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:29.531031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '031e7a50ee4d4ff48d8ad593a924db4e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:29.537912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854d812a-8aa9-4bdd-b82d-095eec07d21b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:29.560266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:29.560933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.562156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.562516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.565638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:29.566117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '031e7a50ee4d4ff48d8ad593a924db4e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:29.567385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:29.567698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '031e7a50ee4d4ff48d8ad593a924db4e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:29.598294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 852/1709] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:29 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:29.605121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:29.611429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.628068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 858/1710] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:29 2022] DELETE /identity/v3/users/031e7a50ee4d4ff48d8ad593a924db4e => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:29.628068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:29.630617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.779892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:29.827306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cfd429-3ff0-46c3-b479-fdb6ebee4493 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:29.842868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:29.847909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:29.848978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:29.849493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ca5550-5b1c-46c4-980e-9edec1e577f9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:29.850588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:29.855426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.878825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:29.881787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:29.898165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 859/1711] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:29 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:29.899605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:29.916434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:29.998941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1317b92-1176-4e34-933d-6a93355ed2b1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:29.999883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1317b92-1176-4e34-933d-6a93355ed2b1 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:30.000116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1317b92-1176-4e34-933d-6a93355ed2b1 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:30.000389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1317b92-1176-4e34-933d-6a93355ed2b1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:30.002218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d1317b92-1176-4e34-933d-6a93355ed2b1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:30.117830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1317b92-1176-4e34-933d-6a93355ed2b1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:30.120327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1317b92-1176-4e34-933d-6a93355ed2b1 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:30.144722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 853/1712] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:29 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 541 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:30.146587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:30.146878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:30.149481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:30.172159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735dcda8-75c7-41af-bd2b-d5c8b9bbe4a1 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:30.188167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f60ac70-3d80-400a-9fd3-bb22c1e837d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:30.189489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f60ac70-3d80-400a-9fd3-bb22c1e837d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:30.195762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f60ac70-3d80-400a-9fd3-bb22c1e837d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:30.196233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f60ac70-3d80-400a-9fd3-bb22c1e837d6 admin admin] PATH_INFO: `/v3/projects/4f49a19ce56d4b3a9641538302a4b291/users/fe055bbd166a4258af2966138765ccae/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:30.205891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:30.207162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:30.232176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 854/1713] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:30.243727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f60ac70-3d80-400a-9fd3-bb22c1e837d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f49a19ce56d4b3a9641538302a4b291, user_id=fe055bbd166a4258af2966138765ccae, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:30.250480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:30.251203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f60ac70-3d80-400a-9fd3-bb22c1e837d6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:30.252300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:30.275438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b67b18-a231-4ee1-bebc-30375e9b4a50 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:30.284756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:30.287066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:30.287908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:30.289568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:30.289822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 860/1714] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:29 2022] PUT /identity/v3/projects/4f49a19ce56d4b3a9641538302a4b291/users/fe055bbd166a4258af2966138765ccae/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:30.305635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:30.309055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:30.310135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:30.318888 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:30.321104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:30.333659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 855/1715] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:32:30.366183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] MFA Rules not processed for user `fe055bbd166a4258af2966138765ccae`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:30.367866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:30.425116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:30.532296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 861/1716] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:30 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:30.537287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:30.552070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e97431-0ab8-488b-b867-3da8abb3db58 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:30.713040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:30.725716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:30.726911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:30.727705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:30.728490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:30.730450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:30.734077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:30.777690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:30.781168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:30.814204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 856/1717] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:30 2022] POST /identity/v3/projects => generated 365 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:30.815591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:30.816510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d25aee-bcbf-4a30-87d5-b939a60fcf37 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:30.985673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797afd5f-2224-4517-acb0-414f4224335a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:31.002162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.002797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.003101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.003851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.004051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.006105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.006281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.006936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.007459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.007743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] PATH_INFO: `/v3/projects/74ed4afcaeed46a3ae1f8ba2c62f1d3f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.014775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] RBAC: Authorizing `identity:delete_project(project_id=74ed4afcaeed46a3ae1f8ba2c62f1d3f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.016858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.062557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 857/1718] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:30 2022] POST /identity/v3/users => generated 289 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:31.064112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:31.099636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf887394-01f8-4d2f-98c9-cbe8d66e6e1e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:31.111563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.113589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.114423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.115171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.122871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.126386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.148777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 858/1719] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:31.148957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74ed4afcaeed46a3ae1f8ba2c62f1d3f', 'request_id': 'req-6b477048-dde7-46ef-9995-2c3b1b22ffdc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.152594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:31.175427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74ed4afcaeed46a3ae1f8ba2c62f1d3f', 'request_id': 'req-6b477048-dde7-46ef-9995-2c3b1b22ffdc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.176959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d7ded-5084-4f86-8dd1-75443a77b14e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:31.208324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74ed4afcaeed46a3ae1f8ba2c62f1d3f', 'request_id': 'req-6b477048-dde7-46ef-9995-2c3b1b22ffdc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.288819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] The token cache is being invalidate because project 74ed4afcaeed46a3ae1f8ba2c62f1d3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:31.294354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.297052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.309554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 862/1720] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:30 2022] DELETE /identity/v3/projects/74ed4afcaeed46a3ae1f8ba2c62f1d3f => generated 0 bytes in 527 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:31.324070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:31.327713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:31.387853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.388476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.388862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.389161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] PATH_INFO: `/v3/projects/b39fa38910814eb285ba6c5831d3ce07/users/4ea7173bf5ad48a3a644966d21dde5a3/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.421221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b39fa38910814eb285ba6c5831d3ce07, user_id=4ea7173bf5ad48a3a644966d21dde5a3, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.426523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.451410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 859/1721] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:31 2022] PUT /identity/v3/projects/b39fa38910814eb285ba6c5831d3ce07/users/4ea7173bf5ad48a3a644966d21dde5a3/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:31.453577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:31.455299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:31.460174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:31.513669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b477048-dde7-46ef-9995-2c3b1b22ffdc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:31.601167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:31.654533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.655228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.655501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.655774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] PATH_INFO: `/v3/users/2ba4589680e74b458ddce087117518ab` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.657748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ba4589680e74b458ddce087117518ab)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.660378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.671914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a834b7-9f73-45f9-af27-7b635b554339 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:31.690145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.691111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.691352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.691610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.698182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:31.729523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.739405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.765834 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 860/1722] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:31.766650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:31.773966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:31.790512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ba4589680e74b458ddce087117518ab'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.792128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76f8073-a401-4541-b5ef-56a39c23d1e9 service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:31.795040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ba4589680e74b458ddce087117518ab'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.800691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.800886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.801217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.801493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.802149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.804265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.819366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ba4589680e74b458ddce087117518ab'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.859563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 861/1723] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:31 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:31.864907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:31.883024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9164a0-294b-43d5-8204-deb06e540464 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:31.883862 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ba4589680e74b458ddce087117518ab'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.888343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ba4589680e74b458ddce087117518ab'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:31.894522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6684c6ca-df63-4f99-82a2-b99b26534045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:31.895448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6684c6ca-df63-4f99-82a2-b99b26534045 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.895776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6684c6ca-df63-4f99-82a2-b99b26534045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.896610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6684c6ca-df63-4f99-82a2-b99b26534045 admin admin] PATH_INFO: `/v3/projects/b39fa38910814eb285ba6c5831d3ce07/users/4ea7173bf5ad48a3a644966d21dde5a3/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.899687 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6684c6ca-df63-4f99-82a2-b99b26534045 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b39fa38910814eb285ba6c5831d3ce07, user_id=4ea7173bf5ad48a3a644966d21dde5a3, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:31.905350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6684c6ca-df63-4f99-82a2-b99b26534045 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:31.923565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 862/1724] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:31 2022] PUT /identity/v3/projects/b39fa38910814eb285ba6c5831d3ce07/users/4ea7173bf5ad48a3a644966d21dde5a3/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:32:31.935260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaef6c24-aac8-47a5-a97a-aa809874b2e6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:31.935686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaef6c24-aac8-47a5-a97a-aa809874b2e6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:31.935962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaef6c24-aac8-47a5-a97a-aa809874b2e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:31.941182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 863/1725] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:31 2022] DELETE /identity/v3/users/2ba4589680e74b458ddce087117518ab => generated 0 bytes in 631 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:32.002059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-eaef6c24-aac8-47a5-a97a-aa809874b2e6 None None] MFA Rules not processed for user `4ea7173bf5ad48a3a644966d21dde5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:32.005682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-eaef6c24-aac8-47a5-a97a-aa809874b2e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:32.221002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 863/1726] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:31 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:32.237710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:32.239563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:32.435242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936aa235-e312-4af9-85b9-c36ddd8eb101 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:32.445568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedeeaa9-2732-401a-b7c6-e3e4adb714d7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:32.446615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedeeaa9-2732-401a-b7c6-e3e4adb714d7 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:32.448016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedeeaa9-2732-401a-b7c6-e3e4adb714d7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:32.449196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedeeaa9-2732-401a-b7c6-e3e4adb714d7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:32.478015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedeeaa9-2732-401a-b7c6-e3e4adb714d7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:32.481273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedeeaa9-2732-401a-b7c6-e3e4adb714d7 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:32.567150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 864/1727] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:32 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:32.706577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaef6c24-aac8-47a5-a97a-aa809874b2e6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:32.724600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaef6c24-aac8-47a5-a97a-aa809874b2e6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:32.733511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abad0bd-73cb-44db-b189-321a6f709912 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:32.735469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abad0bd-73cb-44db-b189-321a6f709912 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:32.735764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abad0bd-73cb-44db-b189-321a6f709912 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:32.738519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abad0bd-73cb-44db-b189-321a6f709912 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:32.745510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8abad0bd-73cb-44db-b189-321a6f709912 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:32.826706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7020986-8deb-4e59-876a-4f0014976bc4 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:32.827251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7020986-8deb-4e59-876a-4f0014976bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:32.828043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7020986-8deb-4e59-876a-4f0014976bc4 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:32.829446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 865/1728] 10.222.0.33 () {54 vars in 912 bytes} [Thu Oct 6 04:32:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:32:32.835122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:32.835908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:32.836049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:32.889797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] MFA Rules not processed for user `87829678318948f3b81bd107076bdfa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:32.891618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:32.918664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abad0bd-73cb-44db-b189-321a6f709912 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:32.923172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abad0bd-73cb-44db-b189-321a6f709912 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:32.945109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 864/1729] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:32 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:32.950252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c223fac-af29-480f-b4f5-5c3a8458b43d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:32.950553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c223fac-af29-480f-b4f5-5c3a8458b43d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:32.950800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c223fac-af29-480f-b4f5-5c3a8458b43d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:32.956931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:33.008113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-8c223fac-af29-480f-b4f5-5c3a8458b43d None None] MFA Rules not processed for user `f68d73b5efc74b3380429cf72e506905`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:33.014253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8c223fac-af29-480f-b4f5-5c3a8458b43d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:33.053006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 866/1730] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:32:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:33.116247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:33.144481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:33.146324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:33.169882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764818a0-6aa7-438f-bf47-6a2a73832684 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:33.180922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:33.182505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:33.182949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:33.183410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:33.201644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:33.205179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:33.254637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 867/1731] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:33.265279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 865/1732] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:33.419397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:33.420848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:33.645229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d39afa4-5842-4643-875e-a8cd761a6484 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:33.648967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab09eeb-4408-44bd-af56-3d5f72b0b940 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:33.652222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab09eeb-4408-44bd-af56-3d5f72b0b940 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:33.654325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab09eeb-4408-44bd-af56-3d5f72b0b940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:33.659670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:33.660328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:33.660678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:33.660858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] PATH_INFO: `/v3/projects/8da5f24e05f04d58a67310fde175410b` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:33.662697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8da5f24e05f04d58a67310fde175410b)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:33.664634 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:33.718433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dab09eeb-4408-44bd-af56-3d5f72b0b940 None None] MFA Rules not processed for user `56c86f73f98a4fb2af787534fd47e2ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:33.719987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dab09eeb-4408-44bd-af56-3d5f72b0b940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:33.811802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8da5f24e05f04d58a67310fde175410b', 'request_id': 'req-638e66cc-41eb-48af-9912-9c8ef33b955c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:33.834013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8da5f24e05f04d58a67310fde175410b', 'request_id': 'req-638e66cc-41eb-48af-9912-9c8ef33b955c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:33.855438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8da5f24e05f04d58a67310fde175410b', 'request_id': 'req-638e66cc-41eb-48af-9912-9c8ef33b955c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:33.897920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] The token cache is being invalidate because project 8da5f24e05f04d58a67310fde175410b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:33.898149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:33.902070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:33.906610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 868/1733] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:33 2022] DELETE /identity/v3/projects/8da5f24e05f04d58a67310fde175410b => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:32:33.908714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:33.912871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:33.945358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 866/1734] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:33.947069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b0a6f-55be-44c2-b6a4-b14767e89cae None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:33.952539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b0a6f-55be-44c2-b6a4-b14767e89cae None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:33.952970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b0a6f-55be-44c2-b6a4-b14767e89cae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:34.019262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-180b0a6f-55be-44c2-b6a4-b14767e89cae None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:34.022011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-180b0a6f-55be-44c2-b6a4-b14767e89cae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:34.153266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:34.201495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e66cc-41eb-48af-9912-9c8ef33b955c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:34.237908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:34.238568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:34.238849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:34.242623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 867/1735] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:34.243546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] PATH_INFO: `/v3/projects/38c90b6829624e059b5ae1afed5a34b9` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:34.245434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] RBAC: Authorizing `identity:delete_project(project_id=38c90b6829624e059b5ae1afed5a34b9)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:34.247018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a54c31-070d-46f1-9c06-b1b2009c14b4 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:34.247308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a54c31-070d-46f1-9c06-b1b2009c14b4 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:34.249177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:34.249512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a54c31-070d-46f1-9c06-b1b2009c14b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:34.300757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-89a54c31-070d-46f1-9c06-b1b2009c14b4 None None] MFA Rules not processed for user `d574235b791e485aa64130c599539945`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:34.307154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-89a54c31-070d-46f1-9c06-b1b2009c14b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:34.389922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38c90b6829624e059b5ae1afed5a34b9', 'request_id': 'req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.414489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38c90b6829624e059b5ae1afed5a34b9', 'request_id': 'req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.429830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38c90b6829624e059b5ae1afed5a34b9', 'request_id': 'req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.476499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 868/1736] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:34 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:34.483838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfe068-de2c-4bd6-a4b4-0dbacdfc7955 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:34.484247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfe068-de2c-4bd6-a4b4-0dbacdfc7955 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:34.484620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfe068-de2c-4bd6-a4b4-0dbacdfc7955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:34.495587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] The token cache is being invalidate because project 38c90b6829624e059b5ae1afed5a34b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:34.495886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.505374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.511725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 869/1737] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:33 2022] DELETE /identity/v3/projects/38c90b6829624e059b5ae1afed5a34b9 => generated 0 bytes in 604 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:34.518097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:34.527044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:34.561905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-52bfe068-de2c-4bd6-a4b4-0dbacdfc7955 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:34.568176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-52bfe068-de2c-4bd6-a4b4-0dbacdfc7955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:34.711925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71e1bd1-b07c-4e9b-a910-dbd0e8adc99a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:34.720361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:34.751217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:34.754629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:34.756656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:34.760427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] PATH_INFO: `/v3/users/43789128dcdf4e838929595a875bf0fe` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:34.764307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=43789128dcdf4e838929595a875bf0fe)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:34.771303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:34.788249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 869/1738] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:34.793316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bfe068-de2c-4bd6-a4b4-0dbacdfc7955 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:34.811260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bfe068-de2c-4bd6-a4b4-0dbacdfc7955 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:34.818136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:34.819511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:34.819511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:34.819791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:34.820616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:34.820948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:34.822993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:34.870574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 870/1739] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:34.872648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:34.912963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac1d1f4-d480-4750-95e4-f454d8dab2e7 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:34.925939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:34.927005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:34.927005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:34.927720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:34.928409 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43789128dcdf4e838929595a875bf0fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.929389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:34.931469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:34.932516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43789128dcdf4e838929595a875bf0fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.945419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43789128dcdf4e838929595a875bf0fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.976738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43789128dcdf4e838929595a875bf0fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:34.980207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43789128dcdf4e838929595a875bf0fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:35.000799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:35.012863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 871/1740] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:34 2022] POST /identity/v3/projects => generated 377 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:35.016641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:35.020765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:35.021280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 870/1741] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:34 2022] DELETE /identity/v3/users/43789128dcdf4e838929595a875bf0fe => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:35.215106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916c9fcf-f2f6-4b95-975e-6d019ecdee82 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:35.324646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:35.325485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:35.325766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:35.326040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:35.326683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:35.329462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:35.407689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 872/1742] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:35 2022] POST /identity/v3/users => generated 289 bytes in 394 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:35.412377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:35.434804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8932c9a-6113-4a21-81a8-36034a4c99ef admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:35.451313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:35.452250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:35.452601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:35.452883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:35.453554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:35.456854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:35.491701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 871/1743] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:35 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:35.493247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:35.518031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca48c6-0094-418e-a92d-5ab378893b7b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:35.538180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279bf38d-a9b7-421d-9ec0-7494158648e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:35.539002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279bf38d-a9b7-421d-9ec0-7494158648e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:35.541818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279bf38d-a9b7-421d-9ec0-7494158648e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:35.542769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279bf38d-a9b7-421d-9ec0-7494158648e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:35.545265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279bf38d-a9b7-421d-9ec0-7494158648e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:35.553489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279bf38d-a9b7-421d-9ec0-7494158648e8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:35.602227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 873/1744] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:35 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:35.602837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:35.621885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bfdce9-d2e4-47fd-af70-7c29941ca2d1 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:35.632002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04766dfe-b442-4765-be3d-0c1ac08b50c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:35.632681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04766dfe-b442-4765-be3d-0c1ac08b50c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:35.635408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04766dfe-b442-4765-be3d-0c1ac08b50c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:35.635695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04766dfe-b442-4765-be3d-0c1ac08b50c2 admin admin] PATH_INFO: `/v3/projects/9abb9325864746039b46c4859b95a237/users/d3e26ea6703e49b1868e1b4220f4dc95/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:35.667911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04766dfe-b442-4765-be3d-0c1ac08b50c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9abb9325864746039b46c4859b95a237, user_id=d3e26ea6703e49b1868e1b4220f4dc95, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:35.678914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04766dfe-b442-4765-be3d-0c1ac08b50c2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:35.724800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 872/1745] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:35 2022] PUT /identity/v3/projects/9abb9325864746039b46c4859b95a237/users/d3e26ea6703e49b1868e1b4220f4dc95/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:32:35.735918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089be56-7a52-44d1-8cbd-b48b87427a26 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:35.736438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089be56-7a52-44d1-8cbd-b48b87427a26 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:35.736870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089be56-7a52-44d1-8cbd-b48b87427a26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:35.879512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f089be56-7a52-44d1-8cbd-b48b87427a26 None None] MFA Rules not processed for user `d3e26ea6703e49b1868e1b4220f4dc95`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:35.883166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f089be56-7a52-44d1-8cbd-b48b87427a26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:35.907545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:36.028168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29db1c-3dfc-43e3-a40c-477e445998b9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:36.028656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29db1c-3dfc-43e3-a40c-477e445998b9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:36.028845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29db1c-3dfc-43e3-a40c-477e445998b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:36.052057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 874/1746] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:35 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:36.053174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f089be56-7a52-44d1-8cbd-b48b87427a26 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:36.067856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9e29db1c-3dfc-43e3-a40c-477e445998b9 None None] MFA Rules not processed for user `a58cd041c02d46dcb8acf6adb11ee9de`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:36.069772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9e29db1c-3dfc-43e3-a40c-477e445998b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:36.264583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 873/1747] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:36 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:36.267910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f089be56-7a52-44d1-8cbd-b48b87427a26 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:36.268642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45784a3d-171a-4ddf-acd0-330238478612 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:36.268942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45784a3d-171a-4ddf-acd0-330238478612 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:36.269221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45784a3d-171a-4ddf-acd0-330238478612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:36.289296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:36.292772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:36.293693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:36.295858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:36.296890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:36.301277 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:36.408689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-45784a3d-171a-4ddf-acd0-330238478612 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:36.414039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-45784a3d-171a-4ddf-acd0-330238478612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:36.438345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 875/1748] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:36 2022] POST /identity/v3/projects => generated 375 bytes in 386 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:36.447617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:36.609642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ff864f-98d9-4102-990f-1c6729933f91 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:36.623563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:36.624219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:36.624501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:36.624772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:36.628185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:36.630441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:36.684267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 876/1749] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:36 2022] POST /identity/v3/users => generated 294 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:36.688339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:36.704212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 874/1750] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:32:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 439 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:36.778262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1189d60-f858-44ef-9a5f-4af5ef0f811d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:36.801420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:36.802523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:36.817921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:36.818285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:36.819120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:36.826258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:36.900739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45784a3d-171a-4ddf-acd0-330238478612 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:36.905173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 877/1751] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:36 2022] GET /identity/v3/roles => generated 1420 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:32:36.922357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45784a3d-171a-4ddf-acd0-330238478612 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:36.967778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:36.968427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:36.968872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:36.969380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] PATH_INFO: `/v3/projects/6aa33f8b563742109cb2609f66b3945a/users/ed1ac85416ae4cf6801db1f6e93e138f/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:36.998709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] RBAC: Authorizing `identity:create_grant(project_id=6aa33f8b563742109cb2609f66b3945a, user_id=ed1ac85416ae4cf6801db1f6e93e138f, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:37.001686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:37.031533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 875/1752] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:36 2022] PUT /identity/v3/projects/6aa33f8b563742109cb2609f66b3945a/users/ed1ac85416ae4cf6801db1f6e93e138f/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:37.035398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:37.055654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:37.059413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:37.062338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eb8812-c9b2-44ab-9c30-935b66940b39 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:37.108538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:37.132912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:37.165182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090d8b7d-aac0-489b-8b70-db6d3cf9526d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:37.171868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:37.173020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:37.173379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:37.173890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] PATH_INFO: `/v3/projects/18db6d5ee5c44bf8bdbd4208c1d7cfca` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:37.175619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=18db6d5ee5c44bf8bdbd4208c1d7cfca)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:37.177505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:37.303931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:37.305850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:37.308369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:37.308818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:37.309635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:37.313782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:37.342762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 878/1753] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:37 2022] GET /identity/v3/roles => generated 1420 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:37.345427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:37.349044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:37.377749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18db6d5ee5c44bf8bdbd4208c1d7cfca', 'request_id': 'req-394cc1e1-87c7-4a66-be17-727d92b954f8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:37.403350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18db6d5ee5c44bf8bdbd4208c1d7cfca', 'request_id': 'req-394cc1e1-87c7-4a66-be17-727d92b954f8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:37.425283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18db6d5ee5c44bf8bdbd4208c1d7cfca', 'request_id': 'req-394cc1e1-87c7-4a66-be17-727d92b954f8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:37.462385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] The token cache is being invalidate because project 18db6d5ee5c44bf8bdbd4208c1d7cfca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:37.469679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:37.472107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:37.477023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 876/1754] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:37 2022] DELETE /identity/v3/projects/18db6d5ee5c44bf8bdbd4208c1d7cfca => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:32:37.477606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:37.485003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:37.776772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4a90d0-a0c7-4867-9194-68060af6fab9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:37.808856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394cc1e1-87c7-4a66-be17-727d92b954f8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:37.840975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b565ee87-8992-4b5e-8a2b-7575f04fcdd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:37.841831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b565ee87-8992-4b5e-8a2b-7575f04fcdd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:37.842114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b565ee87-8992-4b5e-8a2b-7575f04fcdd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:37.842835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b565ee87-8992-4b5e-8a2b-7575f04fcdd9 admin admin] PATH_INFO: `/v3/projects/6aa33f8b563742109cb2609f66b3945a/users/ed1ac85416ae4cf6801db1f6e93e138f/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:37.855892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b565ee87-8992-4b5e-8a2b-7575f04fcdd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6aa33f8b563742109cb2609f66b3945a, user_id=ed1ac85416ae4cf6801db1f6e93e138f, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:37.859029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b565ee87-8992-4b5e-8a2b-7575f04fcdd9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:37.918947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 877/1755] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:37 2022] PUT /identity/v3/projects/6aa33f8b563742109cb2609f66b3945a/users/ed1ac85416ae4cf6801db1f6e93e138f/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:37.920344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e20602-2aa2-405d-a168-3859435388e4 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:37.920625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e20602-2aa2-405d-a168-3859435388e4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:37.920897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e20602-2aa2-405d-a168-3859435388e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:37.996959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-54e20602-2aa2-405d-a168-3859435388e4 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:38.005330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-54e20602-2aa2-405d-a168-3859435388e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:38.049905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27d4a32-818b-4544-8c49-87d7b6eb095d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:38.050527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d4a32-818b-4544-8c49-87d7b6eb095d service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:38.050802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d4a32-818b-4544-8c49-87d7b6eb095d service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:38.051096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d4a32-818b-4544-8c49-87d7b6eb095d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:38.068299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c27d4a32-818b-4544-8c49-87d7b6eb095d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:38.220671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:38.244384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:38.366105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27d4a32-818b-4544-8c49-87d7b6eb095d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:38.367934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27d4a32-818b-4544-8c49-87d7b6eb095d service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:38.395659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 878/1756] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 476 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:38.398077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:38.398460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:38.398827 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:38.402892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 879/1757] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:32:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 1059 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:32:38.405496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e5e9cd-48d4-4756-9b4c-b490d36473e8 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:38.405789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e5e9cd-48d4-4756-9b4c-b490d36473e8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:38.406119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e5e9cd-48d4-4756-9b4c-b490d36473e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:38.521870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e7e5e9cd-48d4-4756-9b4c-b490d36473e8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:38.525049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] MFA Rules not processed for user `ed1ac85416ae4cf6801db1f6e93e138f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:38.526153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e7e5e9cd-48d4-4756-9b4c-b490d36473e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:38.537353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:38.584327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 880/1758] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:38.585307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e5e9cd-48d4-4756-9b4c-b490d36473e8 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:38.646681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e5e9cd-48d4-4756-9b4c-b490d36473e8 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:38.652873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:38.653781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:38.654214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:38.654642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:38.655523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:38.655988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:38.657895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:38.684933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 881/1759] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:38.686477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:38.701730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1d8367-5a15-4b45-a929-cbd84119f702 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:38.719425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:38.720368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:38.721913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:38.722379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:38.723819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:38.727658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:38.812957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 882/1760] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:38 2022] POST /identity/v3/projects => generated 373 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:38.820149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:38.903825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 879/1761] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:38 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 507 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:38.923038 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:38.924909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:39.104603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18597b9-9ed4-40bf-8eb5-18c8f6718508 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:39.115171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982a2f9d-2f4f-47c8-8654-77ac15820de9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:39.131625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:39.152799 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:39.153902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:39.155512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:39.157387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:39.165767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:39.247931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 883/1762] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:38 2022] POST /identity/v3/users => generated 287 bytes in 429 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:32:39.249604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:39.260957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e411cd-1c3f-4215-bb30-0d467e4e4875 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:39.261639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e411cd-1c3f-4215-bb30-0d467e4e4875 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:39.261921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e411cd-1c3f-4215-bb30-0d467e4e4875 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:39.262189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e411cd-1c3f-4215-bb30-0d467e4e4875 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:39.265952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-14e411cd-1c3f-4215-bb30-0d467e4e4875 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:39.271998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e087714e-41f8-4938-a136-775f47bb0218 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:39.284606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:39.285176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:39.288372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:39.288710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:39.289412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:39.291624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:39.346699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 884/1763] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:39 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:39.347530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:39.380567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021aaf22-a6fe-4b6b-b1fa-0376acd3259c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:39.386005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:39.398906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:39.402140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:39.402424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:39.403789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:39.408903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:39.411361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:39.442360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 885/1764] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:39 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:39.445975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:39.460417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e411cd-1c3f-4215-bb30-0d467e4e4875 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:39.463375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e411cd-1c3f-4215-bb30-0d467e4e4875 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:39.479275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468ddd04-9cd2-48bd-b571-3b7ea6731aa3 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:39.497031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:39.497793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:39.502229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 880/1765] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:38 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 577 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:32:39.502364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:39.502364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] PATH_INFO: `/v3/projects/19bb41d03e7c4c3b89724b81dbab4f6c/users/d4c50c9180894fc48f8b71a6cda56482/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:39.550319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19bb41d03e7c4c3b89724b81dbab4f6c, user_id=d4c50c9180894fc48f8b71a6cda56482, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:39.553845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:39.583734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 886/1766] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:39 2022] PUT /identity/v3/projects/19bb41d03e7c4c3b89724b81dbab4f6c/users/d4c50c9180894fc48f8b71a6cda56482/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:39.598093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e131c6bf-dbe4-4144-aed4-b3380a0c064f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:39.598760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e131c6bf-dbe4-4144-aed4-b3380a0c064f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:39.599304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e131c6bf-dbe4-4144-aed4-b3380a0c064f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:39.648719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e131c6bf-dbe4-4144-aed4-b3380a0c064f None None] MFA Rules not processed for user `d4c50c9180894fc48f8b71a6cda56482`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:39.650767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e131c6bf-dbe4-4144-aed4-b3380a0c064f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:39.834098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 881/1767] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:39.874024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:39.876236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:40.015510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:40.068026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff7cdd-2d29-4392-8b70-e6dbd35aac94 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:40.075571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9387531b-df85-4845-8276-7442057803b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:40.076211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9387531b-df85-4845-8276-7442057803b9 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:40.079085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9387531b-df85-4845-8276-7442057803b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:40.079538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9387531b-df85-4845-8276-7442057803b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:40.108194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9387531b-df85-4845-8276-7442057803b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:40.110647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9387531b-df85-4845-8276-7442057803b9 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:40.134075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 887/1768] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:39 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:40.169327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e131c6bf-dbe4-4144-aed4-b3380a0c064f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:40.336574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e131c6bf-dbe4-4144-aed4-b3380a0c064f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:40.347075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:40.347774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:40.348061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:40.348340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] PATH_INFO: `/v3/users/d4c50c9180894fc48f8b71a6cda56482` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:40.349604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4c50c9180894fc48f8b71a6cda56482)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:40.351970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:40.412435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4c50c9180894fc48f8b71a6cda56482'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:40.413353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4c50c9180894fc48f8b71a6cda56482'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:40.425116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4c50c9180894fc48f8b71a6cda56482'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:40.435254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:40.440946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4c50c9180894fc48f8b71a6cda56482'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:40.442229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4c50c9180894fc48f8b71a6cda56482'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:40.513133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:40.513425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:40.513754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:40.523242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 882/1769] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:40 2022] DELETE /identity/v3/users/d4c50c9180894fc48f8b71a6cda56482 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:40.567648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:40.569243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:40.687000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 888/1770] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:40.699749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:40.701672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:40.798434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ae686d-d0e5-4f81-b602-898367218e86 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:40.802566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45b4285-91d4-4852-a1c8-e7257a7b4e26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:40.803344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b4285-91d4-4852-a1c8-e7257a7b4e26 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:40.803689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b4285-91d4-4852-a1c8-e7257a7b4e26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:40.804028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45b4285-91d4-4852-a1c8-e7257a7b4e26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:40.815362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45b4285-91d4-4852-a1c8-e7257a7b4e26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:40.817066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45b4285-91d4-4852-a1c8-e7257a7b4e26 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:40.829687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 883/1771] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:41.488790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:41.491877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:41.520384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbd5fa0-3a2a-4128-9f15-06e89353cb94 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:41.540554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:41.541437 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:41.541835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:41.542171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] PATH_INFO: `/v3/projects/19bb41d03e7c4c3b89724b81dbab4f6c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:41.545625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19bb41d03e7c4c3b89724b81dbab4f6c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:41.548266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:41.656416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19bb41d03e7c4c3b89724b81dbab4f6c', 'request_id': 'req-50868b4f-8c46-4fb2-bae7-2f9e18baf428'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:41.683362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19bb41d03e7c4c3b89724b81dbab4f6c', 'request_id': 'req-50868b4f-8c46-4fb2-bae7-2f9e18baf428'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:41.710129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19bb41d03e7c4c3b89724b81dbab4f6c', 'request_id': 'req-50868b4f-8c46-4fb2-bae7-2f9e18baf428'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:41.733411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] The token cache is being invalidate because project 19bb41d03e7c4c3b89724b81dbab4f6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:41.733870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:41.734892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50868b4f-8c46-4fb2-bae7-2f9e18baf428 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:41.740092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 889/1772] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:41 2022] DELETE /identity/v3/projects/19bb41d03e7c4c3b89724b81dbab4f6c => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:41.750192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a36d1e2-32ba-4f26-af79-a6d92520c555 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:41.750808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a36d1e2-32ba-4f26-af79-a6d92520c555 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:41.750808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a36d1e2-32ba-4f26-af79-a6d92520c555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:41.781643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7a36d1e2-32ba-4f26-af79-a6d92520c555 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:41.783154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7a36d1e2-32ba-4f26-af79-a6d92520c555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:41.998497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 884/1773] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:42.008409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96fd083-191b-43ea-af5a-be80cf326984 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.009035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96fd083-191b-43ea-af5a-be80cf326984 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.010016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96fd083-191b-43ea-af5a-be80cf326984 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.072529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c96fd083-191b-43ea-af5a-be80cf326984 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:42.073939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c96fd083-191b-43ea-af5a-be80cf326984 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:42.111254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 890/1774] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:42.115431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a36d1e2-32ba-4f26-af79-a6d92520c555 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:42.126442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a36d1e2-32ba-4f26-af79-a6d92520c555 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:42.131127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:42.131957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.132398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.132817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.133774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:42.134566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:42.136416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:42.169845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 885/1775] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:42.174384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96fd083-191b-43ea-af5a-be80cf326984 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:42.187115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96fd083-191b-43ea-af5a-be80cf326984 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:42.193551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:42.194177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.194456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.194725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.195409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:42.197746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:42.267238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 891/1776] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:42 2022] POST /identity/v3/projects => generated 375 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:42.267514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:42.284803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd9c8ab-5738-4735-a97a-52154e159cac admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:42.423851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:42.424481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.424756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.425024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.425673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:42.430378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:42.479357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 886/1777] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:42 2022] POST /identity/v3/users => generated 288 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:42.480868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:42.496970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7eef7d9-5240-40a0-a428-f59aaa28159c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:42.511866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:42.512777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.513264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.513738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.514774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:42.518505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:42.564575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 892/1778] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:42 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:42.569041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:42.580596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:42.587067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efaf863c-1122-454a-9bff-8760cd1e093c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:42.605046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933e406-df3d-4fcc-9d03-f10205bdc9dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:42.606206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933e406-df3d-4fcc-9d03-f10205bdc9dd admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.608692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933e406-df3d-4fcc-9d03-f10205bdc9dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.608692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933e406-df3d-4fcc-9d03-f10205bdc9dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.608692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f933e406-df3d-4fcc-9d03-f10205bdc9dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:42.611753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f933e406-df3d-4fcc-9d03-f10205bdc9dd admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:42.635715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 887/1779] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:42 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:42.636850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:42.655082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc840cea-d5b5-4f23-8720-d15833d5c008 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:42.673210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:42.675575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.677814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.680268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] PATH_INFO: `/v3/projects/0a93d5a70da4473089e32d973268d35f/users/baa7f562ab1b401faa4796135e4b1273/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.721740 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a93d5a70da4473089e32d973268d35f, user_id=baa7f562ab1b401faa4796135e4b1273, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:42.732154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:42.756637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:42.776772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 893/1780] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:42 2022] PUT /identity/v3/projects/0a93d5a70da4473089e32d973268d35f/users/baa7f562ab1b401faa4796135e4b1273/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:42.793667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6eb9d-e8fb-4bcf-a6c0-5454882ed572 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:42.795024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6eb9d-e8fb-4bcf-a6c0-5454882ed572 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:42.795304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6eb9d-e8fb-4bcf-a6c0-5454882ed572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:42.868412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3eb6eb9d-e8fb-4bcf-a6c0-5454882ed572 None None] MFA Rules not processed for user `baa7f562ab1b401faa4796135e4b1273`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:42.885023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3eb6eb9d-e8fb-4bcf-a6c0-5454882ed572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:43.059845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 888/1781] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:42 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:43.070475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:43.072818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:43.330498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6945ed39-1084-4453-87ca-6127cd5ff057 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:43.334748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b34c61ebdb9147d5b3d66cbfa556f38e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:43.335842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:43.336274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:43.336696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:43.348070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:43.350087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:43.377733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 894/1782] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:43 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:43.696270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ce804-4dbb-4882-be3d-7786420b94d0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:43.696428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ce804-4dbb-4882-be3d-7786420b94d0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:43.696809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ce804-4dbb-4882-be3d-7786420b94d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:43.789972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9b2ce804-4dbb-4882-be3d-7786420b94d0 None None] MFA Rules not processed for user `87829678318948f3b81bd107076bdfa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:43.795420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9b2ce804-4dbb-4882-be3d-7786420b94d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:43.977190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 889/1783] 10.222.0.25 () {58 vars in 1015 bytes} [Thu Oct 6 04:32:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:44.038091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:44.040313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:44.077893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4f4ec1-9621-4b82-9f12-9a755e9599db service glance] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:44.099615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:44.101348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:44.107489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:44.108591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:44.134181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:44.137176 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:44.174617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 895/1784] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:45.444324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c30babf-fc7f-457b-b6e4-791c7b56e612 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:45.445011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c30babf-fc7f-457b-b6e4-791c7b56e612 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:45.445011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c30babf-fc7f-457b-b6e4-791c7b56e612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:45.502039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6c30babf-fc7f-457b-b6e4-791c7b56e612 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:45.503761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6c30babf-fc7f-457b-b6e4-791c7b56e612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:45.675221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 890/1785] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:32:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:45.692067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:45.695369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:45.933716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce2eed3-7ac5-426b-ab5f-cfb02078d5a9 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:45.943289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aea96a9-50d5-4a78-a131-8092a0485ab9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:45.944361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aea96a9-50d5-4a78-a131-8092a0485ab9 service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:45.944863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aea96a9-50d5-4a78-a131-8092a0485ab9 service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:45.945351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aea96a9-50d5-4a78-a131-8092a0485ab9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:45.960235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aea96a9-50d5-4a78-a131-8092a0485ab9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:45.964799 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aea96a9-50d5-4a78-a131-8092a0485ab9 service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:46.005658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 896/1786] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:32:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:32:46.972286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebe5315-ec4e-4eff-90fc-cfd47f4f04ad None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:46.973277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebe5315-ec4e-4eff-90fc-cfd47f4f04ad None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:46.973757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebe5315-ec4e-4eff-90fc-cfd47f4f04ad None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:46.976926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 891/1787] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:32:46 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:32:46.984638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:46.985140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:46.985218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:47.011009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:47.012013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:47.062746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:47.081694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 897/1788] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:32:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:47.093921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebe5315-ec4e-4eff-90fc-cfd47f4f04ad None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:47.096052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-eebe5315-ec4e-4eff-90fc-cfd47f4f04ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:47.114555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebe5315-ec4e-4eff-90fc-cfd47f4f04ad None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:47.120978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:47.121785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:47.122065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:47.122335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:47.132873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:47.135235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:47.147431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 892/1789] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:53.258443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:53.260128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:53.378268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd3f1e8-bad2-4d24-bc45-9d63179283c6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:53.382408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8729e302-2c84-47e7-9f76-e8c6fc482620 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:53.383245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729e302-2c84-47e7-9f76-e8c6fc482620 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:53.383548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729e302-2c84-47e7-9f76-e8c6fc482620 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:53.383802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8729e302-2c84-47e7-9f76-e8c6fc482620 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:53.386508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8729e302-2c84-47e7-9f76-e8c6fc482620 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:53.528073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8729e302-2c84-47e7-9f76-e8c6fc482620 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:53.530307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8729e302-2c84-47e7-9f76-e8c6fc482620 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:53.556957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 898/1790] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:53 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:54.514999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:54.516430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:54.544673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:54.583816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b7056a-4142-4c29-aea4-5b88cf27b42f service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:54.588374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:54.588999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:54.589277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:54.589547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] PATH_INFO: `/v3/users/fe055bbd166a4258af2966138765ccae` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:54.590666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe055bbd166a4258af2966138765ccae)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:54.592744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:54.683335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe055bbd166a4258af2966138765ccae'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:54.684091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe055bbd166a4258af2966138765ccae'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:54.691326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe055bbd166a4258af2966138765ccae'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:54.704756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe055bbd166a4258af2966138765ccae'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:54.705362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe055bbd166a4258af2966138765ccae'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:54.726507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 893/1791] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:54 2022] DELETE /identity/v3/users/fe055bbd166a4258af2966138765ccae => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:54.731853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:54.732492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:54.732492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:54.752553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:54.753762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:54.841563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 899/1792] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:54.851044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:54.852462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:54.911248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a4973-f73e-435c-a705-f0ef7b915f7a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:54.914825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:54.915597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:54.915736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:54.916009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:54.924769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:54.926180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:54.935983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 894/1793] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:55.234605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:55.236053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:55.250069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee8cef-91d2-4263-9787-ef6121931402 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:55.255699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:55.256486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:55.256763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:55.257032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] PATH_INFO: `/v3/projects/4f49a19ce56d4b3a9641538302a4b291` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:55.258231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f49a19ce56d4b3a9641538302a4b291)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:55.262339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:55.311308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:55.323008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f49a19ce56d4b3a9641538302a4b291', 'request_id': 'req-a0597cee-0ec9-4514-bab8-975366211f4e'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.328468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f49a19ce56d4b3a9641538302a4b291', 'request_id': 'req-a0597cee-0ec9-4514-bab8-975366211f4e'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.334546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f49a19ce56d4b3a9641538302a4b291', 'request_id': 'req-a0597cee-0ec9-4514-bab8-975366211f4e'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.347695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] The token cache is being invalidate because project 4f49a19ce56d4b3a9641538302a4b291 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:55.347981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.348593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.350066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 900/1794] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:55 2022] DELETE /identity/v3/projects/4f49a19ce56d4b3a9641538302a4b291 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:55.354270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:55.363941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba21750-7cbd-4815-9640-0b1417748936 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:55.407352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:55.422149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:55.422775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:55.423085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:55.423359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] PATH_INFO: `/v3/users/4ea7173bf5ad48a3a644966d21dde5a3` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:55.424605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ea7173bf5ad48a3a644966d21dde5a3)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:55.426370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:55.481714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ea7173bf5ad48a3a644966d21dde5a3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.482366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ea7173bf5ad48a3a644966d21dde5a3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.489767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ea7173bf5ad48a3a644966d21dde5a3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.504141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ea7173bf5ad48a3a644966d21dde5a3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.504858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-fba53c2f-acb9-4e30-b8ca-2064770d188e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ea7173bf5ad48a3a644966d21dde5a3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:55.524285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 895/1795] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:55 2022] DELETE /identity/v3/users/4ea7173bf5ad48a3a644966d21dde5a3 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:55.840772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:55.842205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:55.863931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0597cee-0ec9-4514-bab8-975366211f4e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:55.928596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:55.929260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:55.929542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:55.929893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] PATH_INFO: `/v3/projects/b39fa38910814eb285ba6c5831d3ce07` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:55.931991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b39fa38910814eb285ba6c5831d3ce07)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:55.935002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:55.996633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b39fa38910814eb285ba6c5831d3ce07', 'request_id': 'req-e8fd31b0-2b11-4d14-95b2-283c53bab547'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:56.003156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b39fa38910814eb285ba6c5831d3ce07', 'request_id': 'req-e8fd31b0-2b11-4d14-95b2-283c53bab547'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:56.009860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b39fa38910814eb285ba6c5831d3ce07', 'request_id': 'req-e8fd31b0-2b11-4d14-95b2-283c53bab547'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:56.024722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] The token cache is being invalidate because project b39fa38910814eb285ba6c5831d3ce07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:56.025129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:56.025763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e8fd31b0-2b11-4d14-95b2-283c53bab547 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:56.027313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 901/1796] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:55 2022] DELETE /identity/v3/projects/b39fa38910814eb285ba6c5831d3ce07 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:56.032209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21218757-27a4-4eee-b841-fe3914f7fc6b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.032517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21218757-27a4-4eee-b841-fe3914f7fc6b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.032789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21218757-27a4-4eee-b841-fe3914f7fc6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.053730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-21218757-27a4-4eee-b841-fe3914f7fc6b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:56.054946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-21218757-27a4-4eee-b841-fe3914f7fc6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:56.128265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 896/1797] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:32:56.132309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aabd44-59ef-4100-93f5-02218118d064 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.132836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aabd44-59ef-4100-93f5-02218118d064 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.133281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aabd44-59ef-4100-93f5-02218118d064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.155296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-14aabd44-59ef-4100-93f5-02218118d064 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:56.156507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-14aabd44-59ef-4100-93f5-02218118d064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:56.164775 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 902/1798] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:56.169477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21218757-27a4-4eee-b841-fe3914f7fc6b None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.179677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21218757-27a4-4eee-b841-fe3914f7fc6b None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.184410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.185004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.185281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.185549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.186194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:32:56.186497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.188106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.200189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 897/1799] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:32:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:32:56.204519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aabd44-59ef-4100-93f5-02218118d064 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.214616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aabd44-59ef-4100-93f5-02218118d064 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.219399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.220002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.220276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.220545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.221182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.223154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.251607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 903/1800] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:56 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:56.255877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.266061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1ee358-532e-414e-aee7-12adf469385b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.315784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:56.326560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.327360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.327691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.328116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.328810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.330912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.352562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 898/1801] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:56 2022] POST /identity/v3/users => generated 284 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:56.357403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.367862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0a9e3f-fadf-45a4-a273-5f721c54f069 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.372412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.372996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.373272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.373540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.374170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.375806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.385512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 904/1802] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:56.389366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.399205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9232886-d5c5-4a28-b723-5b88cd5e174b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.404371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e4d3f4-35ef-4820-87bf-a555b553236b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.404952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4d3f4-35ef-4820-87bf-a555b553236b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.405229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4d3f4-35ef-4820-87bf-a555b553236b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.405500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4d3f4-35ef-4820-87bf-a555b553236b admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.406132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e4d3f4-35ef-4820-87bf-a555b553236b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.407747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e4d3f4-35ef-4820-87bf-a555b553236b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.418568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 899/1803] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:56.422516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.432748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7a5e5d-1c89-4ad5-bae4-b3c6e68af899 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.437255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.437887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.438162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.438429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] PATH_INFO: `/v3/projects/bcd5ef1be18a4e5c9d5d8f72fb77bbde/users/3b2b2a5851f6448083b0e9b1cb824072/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.450473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:56.454724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcd5ef1be18a4e5c9d5d8f72fb77bbde, user_id=3b2b2a5851f6448083b0e9b1cb824072, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.458221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.472886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 905/1804] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:56 2022] PUT /identity/v3/projects/bcd5ef1be18a4e5c9d5d8f72fb77bbde/users/3b2b2a5851f6448083b0e9b1cb824072/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:56.483654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefb5cdf-6e55-4aac-bbde-9b9da183ed08 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.484169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefb5cdf-6e55-4aac-bbde-9b9da183ed08 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.484169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefb5cdf-6e55-4aac-bbde-9b9da183ed08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.513085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-cefb5cdf-6e55-4aac-bbde-9b9da183ed08 None None] MFA Rules not processed for user `3b2b2a5851f6448083b0e9b1cb824072`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:56.514200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-cefb5cdf-6e55-4aac-bbde-9b9da183ed08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:56.581626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 900/1805] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:56 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:56.586137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.648754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af866f48-3d04-4ee1-be0e-f024f28c502a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.654242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.654850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.655158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.655420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.656063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.657969 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.684450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 906/1806] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:32:56 2022] POST /identity/v3/projects => generated 367 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:56.688636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefb5cdf-6e55-4aac-bbde-9b9da183ed08 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.699932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefb5cdf-6e55-4aac-bbde-9b9da183ed08 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.756166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.756771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.757051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.757324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.757972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.761189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.783459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 901/1807] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:32:56 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:32:56.789922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.800378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b1844-76fb-4ac0-9096-42a9f8454176 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.804898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.805487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.805764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.806032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.806663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.808355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.820953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 907/1808] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:56.825830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.837517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ee55c8-7d4a-473c-85f6-6da7fa0f0fb6 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.842529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.843205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.843485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.843766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] PATH_INFO: `/v3/projects/7f46e47e13c8431b88539776523130e0/users/738cf2a35d5a415c96f8558037c17c47/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.870714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f46e47e13c8431b88539776523130e0, user_id=738cf2a35d5a415c96f8558037c17c47, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.873555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.890237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 902/1809] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:56 2022] PUT /identity/v3/projects/7f46e47e13c8431b88539776523130e0/users/738cf2a35d5a415c96f8558037c17c47/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:56.893896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.905101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53f1fe3-87e1-4db6-b6bf-6f1436b5d752 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.963712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7ae0e2-7813-43d6-8e25-39c7d7d3f47d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.964326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ae0e2-7813-43d6-8e25-39c7d7d3f47d admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.964602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ae0e2-7813-43d6-8e25-39c7d7d3f47d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.964930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ae0e2-7813-43d6-8e25-39c7d7d3f47d admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:56.965731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7ae0e2-7813-43d6-8e25-39c7d7d3f47d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:56.967396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7ae0e2-7813-43d6-8e25-39c7d7d3f47d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:56.978752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 908/1810] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:32:56 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:32:56.982981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:56.993939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafa12b-8c49-46c0-9987-92d024175469 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:56.998440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:56.999185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:56.999461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:56.999798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] PATH_INFO: `/v3/projects/7f46e47e13c8431b88539776523130e0/users/738cf2a35d5a415c96f8558037c17c47/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:57.001700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f46e47e13c8431b88539776523130e0, user_id=738cf2a35d5a415c96f8558037c17c47, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:57.004387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:57.016278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 903/1811] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:32:56 2022] PUT /identity/v3/projects/7f46e47e13c8431b88539776523130e0/users/738cf2a35d5a415c96f8558037c17c47/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:57.026856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bc8d88-0d78-43eb-8736-1c89ef66e99b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:57.027174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bc8d88-0d78-43eb-8736-1c89ef66e99b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:57.027453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bc8d88-0d78-43eb-8736-1c89ef66e99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:57.064379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e2bc8d88-0d78-43eb-8736-1c89ef66e99b None None] MFA Rules not processed for user `738cf2a35d5a415c96f8558037c17c47`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:57.065587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e2bc8d88-0d78-43eb-8736-1c89ef66e99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:57.116317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:57.139616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 909/1812] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:57 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:57.151116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:57.153223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:57.170938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:57.217054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2530c9-1736-4ad2-8af1-e75544c5e222 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:57.220882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796ff3d8-3b72-430e-93e0-f086b80efda3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:57.221600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796ff3d8-3b72-430e-93e0-f086b80efda3 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:57.221873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796ff3d8-3b72-430e-93e0-f086b80efda3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:57.222145 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796ff3d8-3b72-430e-93e0-f086b80efda3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:57.238900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796ff3d8-3b72-430e-93e0-f086b80efda3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:57.240379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796ff3d8-3b72-430e-93e0-f086b80efda3 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:57.252617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 904/1813] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:57 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:32:58.688409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bc8d88-0d78-43eb-8736-1c89ef66e99b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:58.749356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bc8d88-0d78-43eb-8736-1c89ef66e99b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:58.753805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:58.754423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:58.754697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:58.754981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] PATH_INFO: `/v3/users/3b2b2a5851f6448083b0e9b1cb824072` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:58.756090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b2b2a5851f6448083b0e9b1cb824072)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:58.758035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:58.811546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b2b2a5851f6448083b0e9b1cb824072'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:58.812158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b2b2a5851f6448083b0e9b1cb824072'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:58.819408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b2b2a5851f6448083b0e9b1cb824072'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:58.820124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b2b2a5851f6448083b0e9b1cb824072'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:58.832596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b2b2a5851f6448083b0e9b1cb824072'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:58.846794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 910/1814] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:58 2022] DELETE /identity/v3/users/3b2b2a5851f6448083b0e9b1cb824072 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:32:58.851111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:58.851578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:58.851657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:58.872879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:32:58.874217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:58.936518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 905/1815] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:32:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:32:58.946276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:58.947741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:59.005568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28755bd-9e78-49ae-94d9-2a1febf76f9b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:59.009252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:59.009844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:59.010124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:59.010395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:59.019288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:59.020705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:59.030085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 911/1816] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:32:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:32:59.665508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:59.667380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:32:59.688174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb2bfc-5936-4c11-8764-06e72ca81859 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:59.694357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:59.695198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:59.695592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:59.695986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] PATH_INFO: `/v3/projects/bcd5ef1be18a4e5c9d5d8f72fb77bbde` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:59.697882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcd5ef1be18a4e5c9d5d8f72fb77bbde)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:59.699845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:59.752178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcd5ef1be18a4e5c9d5d8f72fb77bbde', 'request_id': 'req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.759621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcd5ef1be18a4e5c9d5d8f72fb77bbde', 'request_id': 'req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.766534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcd5ef1be18a4e5c9d5d8f72fb77bbde', 'request_id': 'req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.779872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] The token cache is being invalidate because project bcd5ef1be18a4e5c9d5d8f72fb77bbde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:32:59.780144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.780910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c0e924c-e01b-4211-b5fe-b5338c6eb2d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.782799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 906/1817] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:32:59 2022] DELETE /identity/v3/projects/bcd5ef1be18a4e5c9d5d8f72fb77bbde => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:32:59.786556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:32:59.797203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc380807-bf9e-4ab2-b7e2-8ac264cfb1e2 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:32:59.800783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:32:59.881308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:32:59.881962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:32:59.882222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:32:59.882539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] PATH_INFO: `/v3/users/738cf2a35d5a415c96f8558037c17c47` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:32:59.884271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=738cf2a35d5a415c96f8558037c17c47)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:32:59.886267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:32:59.971448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '738cf2a35d5a415c96f8558037c17c47'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.973338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '738cf2a35d5a415c96f8558037c17c47'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.990689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '738cf2a35d5a415c96f8558037c17c47'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:32:59.992737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '738cf2a35d5a415c96f8558037c17c47'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:00.025699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5848fce2-1746-4cee-bf7f-a37c3c0e13c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '738cf2a35d5a415c96f8558037c17c47'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:00.064818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 912/1818] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:32:59 2022] DELETE /identity/v3/users/738cf2a35d5a415c96f8558037c17c47 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:01.093891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54239dbd-c082-4d01-ae21-bfcac4371f84 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:01.094342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54239dbd-c082-4d01-ae21-bfcac4371f84 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:01.094342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54239dbd-c082-4d01-ae21-bfcac4371f84 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:01.096818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 907/1819] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:33:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:33:01.263301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:01.264804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:01.265232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:01.301446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:01.303077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:01.429959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 913/1820] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:33:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:01.457034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54239dbd-c082-4d01-ae21-bfcac4371f84 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:01.461355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-54239dbd-c082-4d01-ae21-bfcac4371f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:01.491304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54239dbd-c082-4d01-ae21-bfcac4371f84 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:01.502987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98afc56e-88d1-48a3-8c4e-21bb717bd165 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:01.503826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98afc56e-88d1-48a3-8c4e-21bb717bd165 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:01.504474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98afc56e-88d1-48a3-8c4e-21bb717bd165 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:01.504791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98afc56e-88d1-48a3-8c4e-21bb717bd165 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:01.517864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98afc56e-88d1-48a3-8c4e-21bb717bd165 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:01.519908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98afc56e-88d1-48a3-8c4e-21bb717bd165 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:01.546427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 908/1821] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:01.931472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:01.934027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:02.002870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:02.162203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee41cb8-c0ee-4d5e-802a-91240cce02d4 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:02.169119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:02.171339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.171769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.172171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] PATH_INFO: `/v3/projects/7f46e47e13c8431b88539776523130e0` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.173501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f46e47e13c8431b88539776523130e0)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:02.175626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:02.330390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f46e47e13c8431b88539776523130e0', 'request_id': 'req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:02.339510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f46e47e13c8431b88539776523130e0', 'request_id': 'req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:02.348763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f46e47e13c8431b88539776523130e0', 'request_id': 'req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:02.386209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] The token cache is being invalidate because project 7f46e47e13c8431b88539776523130e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:02.387018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:02.388721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a4cfa93d-66cb-43b3-8700-dacbbfb8cad4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:02.391200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 914/1822] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:01 2022] DELETE /identity/v3/projects/7f46e47e13c8431b88539776523130e0 => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:02.398406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e3c62-2187-40f8-942e-5f27823b02d1 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.399843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e3c62-2187-40f8-942e-5f27823b02d1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.399843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e3c62-2187-40f8-942e-5f27823b02d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.444868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-286e3c62-2187-40f8-942e-5f27823b02d1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:02.446800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-286e3c62-2187-40f8-942e-5f27823b02d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:02.484001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:02.569491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 909/1823] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:02.569959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e16ad-becc-48e0-b9ad-9ba2bba1a8f5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.570244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e16ad-becc-48e0-b9ad-9ba2bba1a8f5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.570711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377e16ad-becc-48e0-b9ad-9ba2bba1a8f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.631869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-377e16ad-becc-48e0-b9ad-9ba2bba1a8f5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:02.633453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-377e16ad-becc-48e0-b9ad-9ba2bba1a8f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:02.649874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 915/1824] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:02.657102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e3c62-2187-40f8-942e-5f27823b02d1 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:02.672242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e3c62-2187-40f8-942e-5f27823b02d1 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:02.678503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:02.679153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.679455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.679767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.680495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:02.681224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:02.682891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:02.698030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 910/1825] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:02.703489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377e16ad-becc-48e0-b9ad-9ba2bba1a8f5 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:02.713536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377e16ad-becc-48e0-b9ad-9ba2bba1a8f5 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:02.718796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:02.719458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.719737 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.719998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.720649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:02.722572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:02.756646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 916/1826] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:02 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:02.758313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:02.769693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6c828d-97fb-4a7b-a85d-a406b77a1c5d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:02.832226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:02.832831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.833276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.833550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.834176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:02.836114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:02.861877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 911/1827] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:02 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:02.863307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:02.874604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fe7ab4-fbf2-41ac-ad5e-2f0b34bf8de0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:02.879219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:02.879821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.880103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.880375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.881017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:02.882773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:02.896492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 917/1828] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:02.898039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:02.908563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb01eb89-03d3-4df7-84cc-be771dd792f5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:02.913150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b94e0c0-e8af-4f48-a3e6-0183909c648c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:02.913758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b94e0c0-e8af-4f48-a3e6-0183909c648c admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.914032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b94e0c0-e8af-4f48-a3e6-0183909c648c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.914299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b94e0c0-e8af-4f48-a3e6-0183909c648c admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.915933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b94e0c0-e8af-4f48-a3e6-0183909c648c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:02.916777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b94e0c0-e8af-4f48-a3e6-0183909c648c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:02.929008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 912/1829] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:02.933963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:02.948368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b111589-7b35-4cae-8c99-88cbed1a8b17 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:02.954432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:02.954832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:02.955251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:02.955617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] PATH_INFO: `/v3/projects/68e06284245b4a7c85ad1eb804bfb2ad/users/122177a569ac475381d720a913b38932/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:02.985394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=68e06284245b4a7c85ad1eb804bfb2ad, user_id=122177a569ac475381d720a913b38932, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:02.988377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.021306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 918/1830] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:02 2022] PUT /identity/v3/projects/68e06284245b4a7c85ad1eb804bfb2ad/users/122177a569ac475381d720a913b38932/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:03.035389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535888aa-3b49-4d9b-862e-4352c637329e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.037758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535888aa-3b49-4d9b-862e-4352c637329e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.037758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535888aa-3b49-4d9b-862e-4352c637329e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.077118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-535888aa-3b49-4d9b-862e-4352c637329e None None] MFA Rules not processed for user `122177a569ac475381d720a913b38932`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:03.078645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-535888aa-3b49-4d9b-862e-4352c637329e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:03.218301 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 913/1831] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:03.229057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:03.291125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:03.320292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca93ee-6eb3-45ef-9fd3-91a31d43df14 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:03.336179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:03.338248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.338681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.339137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.339956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:03.342310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.385112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 919/1832] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:03 2022] POST /identity/v3/projects => generated 383 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:03.386157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535888aa-3b49-4d9b-862e-4352c637329e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:03.398990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535888aa-3b49-4d9b-862e-4352c637329e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:03.422360 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:03.469822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:03.470442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.470718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.471009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.471698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:03.473741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.499199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 914/1833] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:03 2022] POST /identity/v3/users => generated 298 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:03.503377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:03.515172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab1333-378f-4851-9463-e8865779aa75 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:03.519968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:03.520601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.520897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.521147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.521841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:03.523467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.535984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 920/1834] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:33:03.540984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:03.553169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34ed95-efdc-4c8d-a730-8e0b944c5630 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:03.558485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:03.559189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.559511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.559791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] PATH_INFO: `/v3/projects/1216fd88ae284970a7435e711fb4a3cc/users/fb509e12d93542c2852f8d8aece11b74/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.578418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=1216fd88ae284970a7435e711fb4a3cc, user_id=fb509e12d93542c2852f8d8aece11b74, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:03.581540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.599616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 915/1835] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:03 2022] PUT /identity/v3/projects/1216fd88ae284970a7435e711fb4a3cc/users/fb509e12d93542c2852f8d8aece11b74/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:03.604993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:03.617158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cb4824-b405-4f4f-ac5d-42cb12da6d8a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:03.680104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5344de0d-459a-4855-989d-729c123fafe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:03.681239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5344de0d-459a-4855-989d-729c123fafe3 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.681518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5344de0d-459a-4855-989d-729c123fafe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.681821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5344de0d-459a-4855-989d-729c123fafe3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.682513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5344de0d-459a-4855-989d-729c123fafe3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:03.684146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5344de0d-459a-4855-989d-729c123fafe3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.695373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 921/1836] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:03 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:03.700658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:03.711303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a501d35-b3ba-4370-9532-c1985640c6bf admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:03.717255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:03.717914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.718193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.718467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] PATH_INFO: `/v3/projects/1216fd88ae284970a7435e711fb4a3cc/users/fb509e12d93542c2852f8d8aece11b74/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.721269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1216fd88ae284970a7435e711fb4a3cc, user_id=fb509e12d93542c2852f8d8aece11b74, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:03.723442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.736172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 916/1837] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:03 2022] PUT /identity/v3/projects/1216fd88ae284970a7435e711fb4a3cc/users/fb509e12d93542c2852f8d8aece11b74/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:03.748259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098553b4-25ff-40a1-86c6-e0cc0ca2e85c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.748546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098553b4-25ff-40a1-86c6-e0cc0ca2e85c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.748821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098553b4-25ff-40a1-86c6-e0cc0ca2e85c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.782389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-098553b4-25ff-40a1-86c6-e0cc0ca2e85c None None] MFA Rules not processed for user `fb509e12d93542c2852f8d8aece11b74`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:03.784096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-098553b4-25ff-40a1-86c6-e0cc0ca2e85c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:03.864247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 922/1838] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:03 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:03.876514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:03.878460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:03.939707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ecc5b-f2ef-4fba-b58f-32010c68d803 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:03.943487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe92471-e3cf-4605-bbd9-53acaac6a0b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:03.944091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe92471-e3cf-4605-bbd9-53acaac6a0b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:03.944367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe92471-e3cf-4605-bbd9-53acaac6a0b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:03.944638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe92471-e3cf-4605-bbd9-53acaac6a0b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:03.954590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe92471-e3cf-4605-bbd9-53acaac6a0b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:03.955972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe92471-e3cf-4605-bbd9-53acaac6a0b4 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:03.965692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 917/1839] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:03 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:04.034401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098553b4-25ff-40a1-86c6-e0cc0ca2e85c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:04.088681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:04.111945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098553b4-25ff-40a1-86c6-e0cc0ca2e85c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:04.116477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:04.117156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:04.117410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:04.117993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] PATH_INFO: `/v3/users/122177a569ac475381d720a913b38932` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:04.119173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] RBAC: Authorizing `identity:delete_user(user_id=122177a569ac475381d720a913b38932)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:04.121094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:04.188780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '122177a569ac475381d720a913b38932'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.189487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '122177a569ac475381d720a913b38932'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.197996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '122177a569ac475381d720a913b38932'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.198787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '122177a569ac475381d720a913b38932'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.214745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '122177a569ac475381d720a913b38932'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.234429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 923/1840] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:04 2022] DELETE /identity/v3/users/122177a569ac475381d720a913b38932 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:04.238842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:04.239172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:04.239450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:04.262456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:04.263871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:04.340904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 918/1841] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:04.352865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:04.355962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:04.424302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e66a3a9-bd42-4c0e-94d5-2e38fe91a18c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:04.427908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:04.428510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:04.428788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:04.429064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:04.438128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:04.439594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:04.449057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 924/1842] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:04.726632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:04.728445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:04.740709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5f9db-06fd-4e85-8c5c-d0f92488b6ec None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:04.746306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:04.746932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:04.747256 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:04.747535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] PATH_INFO: `/v3/projects/68e06284245b4a7c85ad1eb804bfb2ad` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:04.748657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=68e06284245b4a7c85ad1eb804bfb2ad)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:04.750593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:04.763581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:04.800281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68e06284245b4a7c85ad1eb804bfb2ad', 'request_id': 'req-6c31797f-3d68-4394-9631-c8fe6670cc1c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.807363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68e06284245b4a7c85ad1eb804bfb2ad', 'request_id': 'req-6c31797f-3d68-4394-9631-c8fe6670cc1c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.813480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68e06284245b4a7c85ad1eb804bfb2ad', 'request_id': 'req-6c31797f-3d68-4394-9631-c8fe6670cc1c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.826381 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] The token cache is being invalidate because project 68e06284245b4a7c85ad1eb804bfb2ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:04.826672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.827316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.828800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 919/1843] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:04 2022] DELETE /identity/v3/projects/68e06284245b4a7c85ad1eb804bfb2ad => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:04.832855 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:04.843985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c36cc9-e2a7-4adc-9ba9-418e3d106fd3 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:04.906258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:04.907233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:04.907871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:04.908458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] PATH_INFO: `/v3/users/fb509e12d93542c2852f8d8aece11b74` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:04.909836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb509e12d93542c2852f8d8aece11b74)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:04.911982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:04.916068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:04.995723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb509e12d93542c2852f8d8aece11b74'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:04.996480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb509e12d93542c2852f8d8aece11b74'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.004409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb509e12d93542c2852f8d8aece11b74'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.005189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb509e12d93542c2852f8d8aece11b74'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.018704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-83a26ced-e754-4403-be42-1883d7b022a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb509e12d93542c2852f8d8aece11b74'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.034417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 925/1844] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:04 2022] DELETE /identity/v3/users/fb509e12d93542c2852f8d8aece11b74 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:05.377510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:05.379150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:05.408518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c31797f-3d68-4394-9631-c8fe6670cc1c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:05.496195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:05.496745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.497193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.497642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] PATH_INFO: `/v3/projects/1216fd88ae284970a7435e711fb4a3cc` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.498791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1216fd88ae284970a7435e711fb4a3cc)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:05.500882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:05.551117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1216fd88ae284970a7435e711fb4a3cc', 'request_id': 'req-c6868c10-b641-4e17-ac92-8511e63e7342'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.557993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1216fd88ae284970a7435e711fb4a3cc', 'request_id': 'req-c6868c10-b641-4e17-ac92-8511e63e7342'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.563470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1216fd88ae284970a7435e711fb4a3cc', 'request_id': 'req-c6868c10-b641-4e17-ac92-8511e63e7342'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.576482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] The token cache is being invalidate because project 1216fd88ae284970a7435e711fb4a3cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:05.576819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.577447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c6868c10-b641-4e17-ac92-8511e63e7342 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:05.579049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 920/1845] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:05 2022] DELETE /identity/v3/projects/1216fd88ae284970a7435e711fb4a3cc => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:05.584034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30283515-b5fc-4c0c-883d-8c4befd8f058 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.584513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30283515-b5fc-4c0c-883d-8c4befd8f058 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.584604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30283515-b5fc-4c0c-883d-8c4befd8f058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.604662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-30283515-b5fc-4c0c-883d-8c4befd8f058 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:05.605885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-30283515-b5fc-4c0c-883d-8c4befd8f058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:05.667027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 926/1846] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:05.674817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aa01e2-8269-4a8b-be98-7b441703d733 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.675135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aa01e2-8269-4a8b-be98-7b441703d733 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.675408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aa01e2-8269-4a8b-be98-7b441703d733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.694923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c0aa01e2-8269-4a8b-be98-7b441703d733 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:05.696803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c0aa01e2-8269-4a8b-be98-7b441703d733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:05.704666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 921/1847] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:05.709071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30283515-b5fc-4c0c-883d-8c4befd8f058 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:05.719476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30283515-b5fc-4c0c-883d-8c4befd8f058 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:05.724072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:05.724670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.724945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.725214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.725853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:05.726150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:05.728423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:05.739893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 927/1848] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:05.745024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aa01e2-8269-4a8b-be98-7b441703d733 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:05.754692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aa01e2-8269-4a8b-be98-7b441703d733 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:05.759500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:05.760081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.760353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.760619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.761230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:05.763134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:05.789263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 922/1849] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:05 2022] POST /identity/v3/projects => generated 393 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:05.794089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:05.804038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c80035-ca90-4bdc-ae50-de5d5f025976 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:05.862898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:05.863603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.863881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.864152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.864775 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:05.866696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:05.882376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:05.889941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 928/1850] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:05 2022] POST /identity/v3/users => generated 297 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:05.894235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:05.900432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:05.906067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c8159-7fdb-4dc9-a121-4dce702fa32d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:05.911726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:05.912326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.912781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.913060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.913701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:05.915483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:05.925416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 923/1851] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:05.929551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:05.940157 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ce2ba2-18f9-49aa-8e4b-7a54f2d61c5f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:05.944693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4708be-a81a-4648-9d46-a867ecf3b37c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:05.945276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4708be-a81a-4648-9d46-a867ecf3b37c admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.945553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4708be-a81a-4648-9d46-a867ecf3b37c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.945819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4708be-a81a-4648-9d46-a867ecf3b37c admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.946587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4708be-a81a-4648-9d46-a867ecf3b37c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:05.948238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4708be-a81a-4648-9d46-a867ecf3b37c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:05.958623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 929/1852] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:05 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:05.963569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:05.973657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff9af8f-a954-4c6a-a695-f659dc47e37a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:05.979252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:05.979883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:05.980158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:05.980426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] PATH_INFO: `/v3/projects/4c624641959d461daab575c2bb8b5d20/users/a944522f16c44cbfa933914b52a429fd/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:05.997172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c624641959d461daab575c2bb8b5d20, user_id=a944522f16c44cbfa933914b52a429fd, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.000584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.017437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 924/1853] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:05 2022] PUT /identity/v3/projects/4c624641959d461daab575c2bb8b5d20/users/a944522f16c44cbfa933914b52a429fd/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:06.028479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ec0fc-3cad-48a3-8868-987bdd6c861f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.028808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ec0fc-3cad-48a3-8868-987bdd6c861f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.029224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ec0fc-3cad-48a3-8868-987bdd6c861f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.065452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-719ec0fc-3cad-48a3-8868-987bdd6c861f None None] MFA Rules not processed for user `a944522f16c44cbfa933914b52a429fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:06.066627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-719ec0fc-3cad-48a3-8868-987bdd6c861f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:06.146515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 930/1854] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:06.151848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.221226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d42f8a5-b69a-440a-9bba-e3a039f9181a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.225983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.226585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.226865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.227207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.228028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.229997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.260046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 925/1855] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:06 2022] POST /identity/v3/projects => generated 391 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:06.264132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ec0fc-3cad-48a3-8868-987bdd6c861f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.275043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ec0fc-3cad-48a3-8868-987bdd6c861f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.335663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.336270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.336615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.336950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.337669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.339636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.360802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 931/1856] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:06 2022] POST /identity/v3/users => generated 302 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:06.364840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.376407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5720c2-fd55-4fc3-9432-80da5a56e2c4 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.380967 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.381547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.381818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.382085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.382712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.384362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.395485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 926/1857] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:06.400063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.409987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9989f6-eddf-449a-bfd9-56cd15c1347a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.414560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.415812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.416090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.416363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] PATH_INFO: `/v3/projects/a82498fa56ae4eb3afdb0b6f2cdd3ecf/users/a373a447ac8b4cff9724eb3bb2c29bde/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.434521 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=a82498fa56ae4eb3afdb0b6f2cdd3ecf, user_id=a373a447ac8b4cff9724eb3bb2c29bde, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.437213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.459864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 932/1858] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:06 2022] PUT /identity/v3/projects/a82498fa56ae4eb3afdb0b6f2cdd3ecf/users/a373a447ac8b4cff9724eb3bb2c29bde/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:06.461314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.479435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab99aa0-569d-495e-abb0-301b45be27b9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.552802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2d739a-1e2e-4713-a5ef-c705def52620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.553663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2d739a-1e2e-4713-a5ef-c705def52620 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.553944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2d739a-1e2e-4713-a5ef-c705def52620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.554271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2d739a-1e2e-4713-a5ef-c705def52620 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.554919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2d739a-1e2e-4713-a5ef-c705def52620 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.556661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2d739a-1e2e-4713-a5ef-c705def52620 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.566503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 927/1859] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:06 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:06.570672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.581577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc4b8b4-410d-4d11-882a-fc16698c78bf admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.586206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.587500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.587780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.588328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] PATH_INFO: `/v3/projects/a82498fa56ae4eb3afdb0b6f2cdd3ecf/users/a373a447ac8b4cff9724eb3bb2c29bde/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.590044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a82498fa56ae4eb3afdb0b6f2cdd3ecf, user_id=a373a447ac8b4cff9724eb3bb2c29bde, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.592787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.603626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 933/1860] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:06 2022] PUT /identity/v3/projects/a82498fa56ae4eb3afdb0b6f2cdd3ecf/users/a373a447ac8b4cff9724eb3bb2c29bde/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:06.615912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.616227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.616564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.655005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] MFA Rules not processed for user `a373a447ac8b4cff9724eb3bb2c29bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:06.655865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:06.684459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:06.731375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 928/1861] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:06 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:06.741949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.744687 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:06.788436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:06.815707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7eca1e-8e66-4528-8f93-78542fdcea69 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.819507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.820117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.820397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.820703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.831630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.833096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.843134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 934/1862] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:06 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:06.892050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:06.893445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:06.905771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eaa974-0993-4af7-8c9c-adfb56c46e45 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:06.909661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7874296c-809b-4fdb-8290-4aa2868cc47c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:06.910248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7874296c-809b-4fdb-8290-4aa2868cc47c service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:06.910535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7874296c-809b-4fdb-8290-4aa2868cc47c service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:06.910806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7874296c-809b-4fdb-8290-4aa2868cc47c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:06.974334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7874296c-809b-4fdb-8290-4aa2868cc47c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:06.975784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7874296c-809b-4fdb-8290-4aa2868cc47c service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:06.984669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 929/1863] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:06 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:33:10.574900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:10.650524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630442ec-d6f1-4192-aa32-e7a319710d44 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:10.657162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:10.657684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:10.657834 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:10.658137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] PATH_INFO: `/v3/users/a944522f16c44cbfa933914b52a429fd` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:10.659362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=a944522f16c44cbfa933914b52a429fd)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:10.661733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:10.752156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a944522f16c44cbfa933914b52a429fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:10.752979 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a944522f16c44cbfa933914b52a429fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:10.761755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a944522f16c44cbfa933914b52a429fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:10.762348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a944522f16c44cbfa933914b52a429fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:10.774641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a944522f16c44cbfa933914b52a429fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:10.788982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 935/1864] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:10 2022] DELETE /identity/v3/users/a944522f16c44cbfa933914b52a429fd => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:10.793469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:10.794003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:10.794094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:10.814711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:10.816481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:10.892670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 930/1865] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:10.902432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:10.904132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:10.974883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26df7b1a-ebf0-423b-9a9b-75adc2a1f9fc admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:10.978480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:10.979117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:10.979395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:10.979662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:10.985670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:10.991581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:10.993747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:11.007041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 936/1866] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:11.384826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:11.386296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:11.398332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6762ebc-5dd8-44c8-870c-65efe45f06ed None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:11.404764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:11.405412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:11.405686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:11.405980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] PATH_INFO: `/v3/projects/4c624641959d461daab575c2bb8b5d20` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:11.407215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c624641959d461daab575c2bb8b5d20)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:11.409217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:11.456898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c624641959d461daab575c2bb8b5d20', 'request_id': 'req-d89d71bc-2575-4bca-accc-db936ed30cdb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.463871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c624641959d461daab575c2bb8b5d20', 'request_id': 'req-d89d71bc-2575-4bca-accc-db936ed30cdb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.465795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:11.470696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c624641959d461daab575c2bb8b5d20', 'request_id': 'req-d89d71bc-2575-4bca-accc-db936ed30cdb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.483890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] The token cache is being invalidate because project 4c624641959d461daab575c2bb8b5d20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:11.484175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.484787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.486272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 931/1867] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:11 2022] DELETE /identity/v3/projects/4c624641959d461daab575c2bb8b5d20 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:11.490197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:11.500342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d32805a-68d3-439f-8e5e-87dbd2128e4e service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:11.561871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:11.562508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:11.562789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:11.563100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] PATH_INFO: `/v3/users/a373a447ac8b4cff9724eb3bb2c29bde` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:11.564414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a373a447ac8b4cff9724eb3bb2c29bde)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:11.566352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:11.618441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a373a447ac8b4cff9724eb3bb2c29bde'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.619107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a373a447ac8b4cff9724eb3bb2c29bde'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.627037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a373a447ac8b4cff9724eb3bb2c29bde'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.627733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a373a447ac8b4cff9724eb3bb2c29bde'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.643770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c24987d6-2b15-47dc-a9f1-25e57571ef1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a373a447ac8b4cff9724eb3bb2c29bde'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:11.664268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 937/1868] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:11 2022] DELETE /identity/v3/users/a373a447ac8b4cff9724eb3bb2c29bde => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:11.967573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:11.969245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:11.983211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89d71bc-2575-4bca-accc-db936ed30cdb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.041239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.041866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.042222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.042494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] PATH_INFO: `/v3/projects/a82498fa56ae4eb3afdb0b6f2cdd3ecf` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.043628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a82498fa56ae4eb3afdb0b6f2cdd3ecf)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.045523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.090547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a82498fa56ae4eb3afdb0b6f2cdd3ecf', 'request_id': 'req-1516bc7b-ea4a-454b-b568-31be47de1b9d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:12.096579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a82498fa56ae4eb3afdb0b6f2cdd3ecf', 'request_id': 'req-1516bc7b-ea4a-454b-b568-31be47de1b9d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:12.102663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a82498fa56ae4eb3afdb0b6f2cdd3ecf', 'request_id': 'req-1516bc7b-ea4a-454b-b568-31be47de1b9d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:12.120138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] The token cache is being invalidate because project a82498fa56ae4eb3afdb0b6f2cdd3ecf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:12.120695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:12.121537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1516bc7b-ea4a-454b-b568-31be47de1b9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:12.123323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 932/1869] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:11 2022] DELETE /identity/v3/projects/a82498fa56ae4eb3afdb0b6f2cdd3ecf => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:12.128186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46308d53-13f7-462b-8e6e-fa56325daa29 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.128491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46308d53-13f7-462b-8e6e-fa56325daa29 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.128764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46308d53-13f7-462b-8e6e-fa56325daa29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.148545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-46308d53-13f7-462b-8e6e-fa56325daa29 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:12.150381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-46308d53-13f7-462b-8e6e-fa56325daa29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:12.235213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 938/1870] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:12.236846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37ed18-5713-46fe-aeb3-ce0d234867d3 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.237132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37ed18-5713-46fe-aeb3-ce0d234867d3 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.237403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37ed18-5713-46fe-aeb3-ce0d234867d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.259914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3c37ed18-5713-46fe-aeb3-ce0d234867d3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:12.261166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3c37ed18-5713-46fe-aeb3-ce0d234867d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:12.271219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 933/1871] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:12.279711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46308d53-13f7-462b-8e6e-fa56325daa29 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.289431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46308d53-13f7-462b-8e6e-fa56325daa29 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.294551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.295236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.295512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.295786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.298512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:12.298512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.298512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.309303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:12.315533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 939/1872] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:12.321465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37ed18-5713-46fe-aeb3-ce0d234867d3 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.334104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37ed18-5713-46fe-aeb3-ce0d234867d3 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.339265 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.339962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.340306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.341228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.341944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.344030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.376067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 934/1873] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:12 2022] POST /identity/v3/projects => generated 381 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:12.377301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.387065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e56654e-e88d-495a-8794-47de988e47c7 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.445063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.445734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.445955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.446229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.446862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.448823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.473837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 940/1874] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:12 2022] POST /identity/v3/users => generated 291 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:33:12.476625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.487471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc44429-94e9-418f-b231-9d47acd0ace9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.492080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.493242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.493527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.493798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.494439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.496051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.506302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 935/1875] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:33:12.511095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.521221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5331ebd-d1fe-4d34-a69d-d917234703f3 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.525967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda971e3-edd5-4683-a9cc-5fdf44a16e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.526556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda971e3-edd5-4683-a9cc-5fdf44a16e74 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.526832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda971e3-edd5-4683-a9cc-5fdf44a16e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.527142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda971e3-edd5-4683-a9cc-5fdf44a16e74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.527777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda971e3-edd5-4683-a9cc-5fdf44a16e74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.529388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda971e3-edd5-4683-a9cc-5fdf44a16e74 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.540309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 941/1876] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:12.544504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.554570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8906aa-2bac-4409-8eff-0085194ffdd8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.560100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.560736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.561012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.561282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] PATH_INFO: `/v3/projects/ced034b88d4043b0bc87a4b70a644874/users/a5d78b7a0f2644db9bc6c3aec02fca28/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.576597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ced034b88d4043b0bc87a4b70a644874, user_id=a5d78b7a0f2644db9bc6c3aec02fca28, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.579233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.596501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 936/1877] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:12 2022] PUT /identity/v3/projects/ced034b88d4043b0bc87a4b70a644874/users/a5d78b7a0f2644db9bc6c3aec02fca28/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:12.607414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a236fb6b-53a5-4602-b023-47a9401acd9a None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.607698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a236fb6b-53a5-4602-b023-47a9401acd9a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.607967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a236fb6b-53a5-4602-b023-47a9401acd9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.637558 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a236fb6b-53a5-4602-b023-47a9401acd9a None None] MFA Rules not processed for user `a5d78b7a0f2644db9bc6c3aec02fca28`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:12.638767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a236fb6b-53a5-4602-b023-47a9401acd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:12.703756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 942/1878] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:12 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:12.708141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.721505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:12.777866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c88f00-db7f-4996-bcff-21ab3fe77332 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.782705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.783344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.783624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.783891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.784535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.786507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.816656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 937/1879] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:12 2022] POST /identity/v3/projects => generated 383 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:12.820937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a236fb6b-53a5-4602-b023-47a9401acd9a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.832547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a236fb6b-53a5-4602-b023-47a9401acd9a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.908167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.908784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.909113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.909384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.910208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.921668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.947343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:12.955560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 943/1880] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:12 2022] POST /identity/v3/users => generated 298 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:12.959779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:12.971850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cc4e40-3fdc-458d-b613-397694a19139 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:12.976811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:12.978335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:12.978615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:12.979044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:12.979694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:12.981369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:12.994988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 938/1881] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:12.997117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:13.006996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56809f27-5661-41fd-a276-3cc9989c777f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:13.012269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:13.012913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:13.013193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:13.013467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] PATH_INFO: `/v3/projects/fa0361c43819425a899e3901c2dd95da/users/1dd5f0195fad4f9ab1dbc3372819f6a6/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:13.028956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa0361c43819425a899e3901c2dd95da, user_id=1dd5f0195fad4f9ab1dbc3372819f6a6, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:13.031790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:13.044592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 944/1882] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:12 2022] PUT /identity/v3/projects/fa0361c43819425a899e3901c2dd95da/users/1dd5f0195fad4f9ab1dbc3372819f6a6/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:13.048560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:13.058355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549798d-c15d-4d98-9c19-0c3f92240fab admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:13.113297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817ac0e4-eaf5-4ba3-9aae-a9ac1e3b6211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:13.113887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ac0e4-eaf5-4ba3-9aae-a9ac1e3b6211 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:13.114161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ac0e4-eaf5-4ba3-9aae-a9ac1e3b6211 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:13.114433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ac0e4-eaf5-4ba3-9aae-a9ac1e3b6211 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:13.115090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817ac0e4-eaf5-4ba3-9aae-a9ac1e3b6211 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:13.117086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817ac0e4-eaf5-4ba3-9aae-a9ac1e3b6211 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:13.127236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 939/1883] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:13 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:13.131476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:13.144472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a024c5e-6614-4ebf-93ed-5fd605047ce9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:13.149501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:13.150192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:13.150467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:13.150736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] PATH_INFO: `/v3/projects/fa0361c43819425a899e3901c2dd95da/users/1dd5f0195fad4f9ab1dbc3372819f6a6/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:13.152784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa0361c43819425a899e3901c2dd95da, user_id=1dd5f0195fad4f9ab1dbc3372819f6a6, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:13.155471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:13.166593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 945/1884] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:13 2022] PUT /identity/v3/projects/fa0361c43819425a899e3901c2dd95da/users/1dd5f0195fad4f9ab1dbc3372819f6a6/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:13.177518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11594f03-0bde-40d9-b592-6ddfa19be0fe None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:13.177803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11594f03-0bde-40d9-b592-6ddfa19be0fe None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:13.178076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11594f03-0bde-40d9-b592-6ddfa19be0fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:13.206282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-11594f03-0bde-40d9-b592-6ddfa19be0fe None None] MFA Rules not processed for user `1dd5f0195fad4f9ab1dbc3372819f6a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:13.207482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-11594f03-0bde-40d9-b592-6ddfa19be0fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:13.278575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 940/1885] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:13 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:13.291621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:13.293122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:13.355030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5399a50a-5084-4162-825c-d50dde6e0f4a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:13.358674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:13.359406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:13.359735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:13.360002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:13.369655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:13.371215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:13.381276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 946/1886] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:13 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:13.474090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11594f03-0bde-40d9-b592-6ddfa19be0fe None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:13.483776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11594f03-0bde-40d9-b592-6ddfa19be0fe None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:13.487445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacacbee-1ef9-4c52-bec0-a3e33a95811c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:13.488113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacacbee-1ef9-4c52-bec0-a3e33a95811c service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:13.488457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacacbee-1ef9-4c52-bec0-a3e33a95811c service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:13.488793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacacbee-1ef9-4c52-bec0-a3e33a95811c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:13.493773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:13.551908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacacbee-1ef9-4c52-bec0-a3e33a95811c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:13.553368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacacbee-1ef9-4c52-bec0-a3e33a95811c service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:13.563588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 941/1887] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:13 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:13.850722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:13.912314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a076bb-77a2-4c24-93dc-2ed1370907f6 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:13.917332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:13.917942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:13.918215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:13.918480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] PATH_INFO: `/v3/users/a5d78b7a0f2644db9bc6c3aec02fca28` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:13.919596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5d78b7a0f2644db9bc6c3aec02fca28)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:13.921488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:13.982937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5d78b7a0f2644db9bc6c3aec02fca28'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:13.983678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5d78b7a0f2644db9bc6c3aec02fca28'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:13.991271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5d78b7a0f2644db9bc6c3aec02fca28'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:13.992198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5d78b7a0f2644db9bc6c3aec02fca28'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.002292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:14.011050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5d78b7a0f2644db9bc6c3aec02fca28'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.036979 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 947/1888] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:13 2022] DELETE /identity/v3/users/a5d78b7a0f2644db9bc6c3aec02fca28 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:14.043331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:14.044356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:14.044633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:14.069655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:14.070906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:14.136324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 942/1889] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:14.145603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:14.147056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:14.208945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd5479-d0df-490b-82aa-7e50abb97a80 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:14.214401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:14.215005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:14.215294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:14.215568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:14.225688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:14.228371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:14.238389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 948/1890] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:14.517895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:14.519434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:14.534870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e756339-e343-430a-b73f-81dbccd934ef None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:14.540866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:14.541752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:14.542118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:14.543319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] PATH_INFO: `/v3/projects/ced034b88d4043b0bc87a4b70a644874` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:14.546067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ced034b88d4043b0bc87a4b70a644874)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:14.550050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:14.600709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ced034b88d4043b0bc87a4b70a644874', 'request_id': 'req-3530c053-fb6a-4986-8e21-6e64af00e3b9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.607526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ced034b88d4043b0bc87a4b70a644874', 'request_id': 'req-3530c053-fb6a-4986-8e21-6e64af00e3b9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.613826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ced034b88d4043b0bc87a4b70a644874', 'request_id': 'req-3530c053-fb6a-4986-8e21-6e64af00e3b9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.628903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] The token cache is being invalidate because project ced034b88d4043b0bc87a4b70a644874 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:14.631258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.631258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.631659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 943/1891] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:14 2022] DELETE /identity/v3/projects/ced034b88d4043b0bc87a4b70a644874 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:14.636449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:14.645417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e5280-72e0-45de-b3ee-fb257be5c242 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:14.704940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:14.705567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:14.705840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:14.706134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] PATH_INFO: `/v3/users/1dd5f0195fad4f9ab1dbc3372819f6a6` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:14.707272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dd5f0195fad4f9ab1dbc3372819f6a6)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:14.709183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:14.758178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dd5f0195fad4f9ab1dbc3372819f6a6'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.759018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dd5f0195fad4f9ab1dbc3372819f6a6'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.765816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dd5f0195fad4f9ab1dbc3372819f6a6'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.766450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dd5f0195fad4f9ab1dbc3372819f6a6'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.778463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6be4445e-a0a7-4b85-a005-f967c22fc488 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dd5f0195fad4f9ab1dbc3372819f6a6'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:14.792271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 949/1892] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:14 2022] DELETE /identity/v3/users/1dd5f0195fad4f9ab1dbc3372819f6a6 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:15.064882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.066408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:15.103752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3530c053-fb6a-4986-8e21-6e64af00e3b9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.150029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:15.185261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.185931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.186211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.186859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] PATH_INFO: `/v3/projects/fa0361c43819425a899e3901c2dd95da` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.187650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa0361c43819425a899e3901c2dd95da)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.189654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.269134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa0361c43819425a899e3901c2dd95da', 'request_id': 'req-8f280e5a-3ce2-4ff4-b125-751d0fa96302'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:15.277899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa0361c43819425a899e3901c2dd95da', 'request_id': 'req-8f280e5a-3ce2-4ff4-b125-751d0fa96302'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:15.284567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa0361c43819425a899e3901c2dd95da', 'request_id': 'req-8f280e5a-3ce2-4ff4-b125-751d0fa96302'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:15.297983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] The token cache is being invalidate because project fa0361c43819425a899e3901c2dd95da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:15.298519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:15.299085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8f280e5a-3ce2-4ff4-b125-751d0fa96302 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:15.300897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 944/1893] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:15 2022] DELETE /identity/v3/projects/fa0361c43819425a899e3901c2dd95da => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:15.309088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084832e-f89e-435a-b2df-db580939d4a3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.309632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084832e-f89e-435a-b2df-db580939d4a3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.310289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084832e-f89e-435a-b2df-db580939d4a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.319726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:15.333474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3084832e-f89e-435a-b2df-db580939d4a3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:15.335118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3084832e-f89e-435a-b2df-db580939d4a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:15.409765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 950/1894] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:15.417897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9583e7f-ee5b-4c66-840a-4c60b358898c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.418185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9583e7f-ee5b-4c66-840a-4c60b358898c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.418462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9583e7f-ee5b-4c66-840a-4c60b358898c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.440454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b9583e7f-ee5b-4c66-840a-4c60b358898c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:15.441663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b9583e7f-ee5b-4c66-840a-4c60b358898c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:15.449850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 945/1895] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:15.453857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3084832e-f89e-435a-b2df-db580939d4a3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.463625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3084832e-f89e-435a-b2df-db580939d4a3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.468245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.469956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.469956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.469956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.470335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:15.470505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.472173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.486926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 951/1896] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:15.496647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9583e7f-ee5b-4c66-840a-4c60b358898c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.507558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9583e7f-ee5b-4c66-840a-4c60b358898c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.512871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.513472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.513748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.514018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.514649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.516890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.550566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 946/1897] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:15 2022] POST /identity/v3/projects => generated 371 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:15.554633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.565763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea5a8c-efee-452c-a7a0-e39715b4573b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.634003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.634614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.634884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.635184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.635829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.638470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.658712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 952/1898] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:15 2022] POST /identity/v3/users => generated 286 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:33:15.662951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.673318 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77940ff9-bd75-42a2-9ba7-76ce54d61b6c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.677881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.678466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.678742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.679044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.679678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.682064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.692254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 947/1899] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:15.696874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.706711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2393dc42-ad8d-4abb-b4e5-88ef65b42c49 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.711292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2989118c-e3a3-4fca-8eda-b574833c688b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.711889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2989118c-e3a3-4fca-8eda-b574833c688b admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.712162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2989118c-e3a3-4fca-8eda-b574833c688b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.712433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2989118c-e3a3-4fca-8eda-b574833c688b admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.713077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2989118c-e3a3-4fca-8eda-b574833c688b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.715086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2989118c-e3a3-4fca-8eda-b574833c688b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.725628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 953/1900] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:15.729685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.739405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc28923-c848-40aa-bbef-f9d0db7a1e41 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.744461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.745096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.745366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.745634 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] PATH_INFO: `/v3/projects/a5333719f29749769b2f1052e453d4dd/users/e4598fe8b3364ca0b3a818edd0a54ff2/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.760622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5333719f29749769b2f1052e453d4dd, user_id=e4598fe8b3364ca0b3a818edd0a54ff2, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.763191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.777141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 948/1901] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:15 2022] PUT /identity/v3/projects/a5333719f29749769b2f1052e453d4dd/users/e4598fe8b3364ca0b3a818edd0a54ff2/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:15.782272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13647580-4778-4522-9996-95a17d5f7be8 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.782626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13647580-4778-4522-9996-95a17d5f7be8 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.783025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13647580-4778-4522-9996-95a17d5f7be8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.801809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-13647580-4778-4522-9996-95a17d5f7be8 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:15.803163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-13647580-4778-4522-9996-95a17d5f7be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:15.866422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 954/1902] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:15.877232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:15.878982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:15.941087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a093ede-2df0-4b88-8aae-d0db2987de98 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:15.944855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:15.945451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:15.945873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:15.946126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:15.955548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:15.956971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:15.961193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:15.970367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 949/1903] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:16.839400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f80846-a991-4528-b689-09769661ec80 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:16.839988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f80846-a991-4528-b689-09769661ec80 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:16.839988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f80846-a991-4528-b689-09769661ec80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:16.861002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:16.871275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c4f80846-a991-4528-b689-09769661ec80 None None] MFA Rules not processed for user `e4598fe8b3364ca0b3a818edd0a54ff2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:16.872503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c4f80846-a991-4528-b689-09769661ec80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:16.969936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 955/1904] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:16 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:16.974467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:16.985456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad739433-c4d6-440c-97a4-c1570b77d715 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:16.990248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:16.990872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:16.991873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:16.991873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:16.992053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:16.994249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:17.027439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 950/1905] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:16 2022] POST /identity/v3/projects => generated 373 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:17.032640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f80846-a991-4528-b689-09769661ec80 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:17.045236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f80846-a991-4528-b689-09769661ec80 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:17.108058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:17.108664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:17.108942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:17.109212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:17.109836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:17.111858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:17.133055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 956/1906] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:17 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:17.137053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:17.147586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168c73da-7d15-4bc2-95c3-4fc1c567bc3c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:17.152998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:17.153590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:17.153969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:17.154250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:17.154944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:17.156843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:17.179426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 951/1907] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:17 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:17.183535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:17.193707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe154ef0-c6f7-4a8c-811b-0e6afa353896 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:17.198515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:17.199218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:17.199498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:17.199769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] PATH_INFO: `/v3/projects/531d455406924a40b95f3de63b192e9a/users/b85310a65d03474c93225c8d31e6a730/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:17.223390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=531d455406924a40b95f3de63b192e9a, user_id=b85310a65d03474c93225c8d31e6a730, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:17.227710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:17.243778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 957/1908] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:17 2022] PUT /identity/v3/projects/531d455406924a40b95f3de63b192e9a/users/b85310a65d03474c93225c8d31e6a730/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:17.248148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:17.259486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54672cbe-d9fe-4555-8854-0f9121ead768 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:17.319282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eba17d3-6d87-414f-b01e-51f6722ae794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:17.319970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba17d3-6d87-414f-b01e-51f6722ae794 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:17.320306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba17d3-6d87-414f-b01e-51f6722ae794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:17.321345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba17d3-6d87-414f-b01e-51f6722ae794 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:17.322056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eba17d3-6d87-414f-b01e-51f6722ae794 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:17.323870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eba17d3-6d87-414f-b01e-51f6722ae794 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:17.335018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 952/1909] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:17 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:17.338450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:17.349001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1540bc96-55fd-4f2a-a02a-bf408f9339a2 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:17.353704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a977c9a-db6f-4c98-86b6-d343d07e72df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:17.354356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a977c9a-db6f-4c98-86b6-d343d07e72df admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:17.354639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a977c9a-db6f-4c98-86b6-d343d07e72df admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:17.354910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a977c9a-db6f-4c98-86b6-d343d07e72df admin admin] PATH_INFO: `/v3/projects/531d455406924a40b95f3de63b192e9a/users/b85310a65d03474c93225c8d31e6a730/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:17.357157 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a977c9a-db6f-4c98-86b6-d343d07e72df admin admin] RBAC: Authorizing `identity:create_grant(project_id=531d455406924a40b95f3de63b192e9a, user_id=b85310a65d03474c93225c8d31e6a730, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:17.359855 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a977c9a-db6f-4c98-86b6-d343d07e72df admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:17.371508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 958/1910] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:17 2022] PUT /identity/v3/projects/531d455406924a40b95f3de63b192e9a/users/b85310a65d03474c93225c8d31e6a730/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:17.974737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cc5019-6b16-44dc-ae6a-785e000bc1be None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:17.975686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cc5019-6b16-44dc-ae6a-785e000bc1be None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:17.976421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cc5019-6b16-44dc-ae6a-785e000bc1be None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:17.979051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 953/1911] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:33:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:33:17.992348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee09da0-9afb-4902-a52b-904818395d65 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:17.995280 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee09da0-9afb-4902-a52b-904818395d65 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:18.001885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee09da0-9afb-4902-a52b-904818395d65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:18.032833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-7ee09da0-9afb-4902-a52b-904818395d65 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:18.034069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7ee09da0-9afb-4902-a52b-904818395d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:18.093360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:18.118082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 959/1912] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:33:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:18.130412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cc5019-6b16-44dc-ae6a-785e000bc1be None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:18.132690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e5cc5019-6b16-44dc-ae6a-785e000bc1be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:18.156482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cc5019-6b16-44dc-ae6a-785e000bc1be None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:18.162951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:18.163752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:18.164091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:18.164418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:18.178691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:18.188434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:18.213748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 954/1913] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:18.654883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24220f-d4ff-47f5-866d-bf3e3562343c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:18.655192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24220f-d4ff-47f5-866d-bf3e3562343c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:18.655460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f24220f-d4ff-47f5-866d-bf3e3562343c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:18.715460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-7f24220f-d4ff-47f5-866d-bf3e3562343c None None] MFA Rules not processed for user `b85310a65d03474c93225c8d31e6a730`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:18.716696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7f24220f-d4ff-47f5-866d-bf3e3562343c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:18.803126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 960/1914] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:18 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:18.815640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:18.818408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:18.918908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a0c44e-f2ad-47d8-828a-8f52b96c312b service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:18.928830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:18.933052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:18.935554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:18.936410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:19.043333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:19.046069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:19.058224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 955/1915] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:18 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:24.812296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f24220f-d4ff-47f5-866d-bf3e3562343c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:24.821904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f24220f-d4ff-47f5-866d-bf3e3562343c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:24.825450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd75358-a1b0-40c3-9a3e-2956236c9b24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:24.826061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd75358-a1b0-40c3-9a3e-2956236c9b24 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:24.826317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd75358-a1b0-40c3-9a3e-2956236c9b24 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:24.826586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd75358-a1b0-40c3-9a3e-2956236c9b24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:24.835334 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd75358-a1b0-40c3-9a3e-2956236c9b24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:24.837265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd75358-a1b0-40c3-9a3e-2956236c9b24 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:24.846570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 961/1916] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:24 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:27.840120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:27.842674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:27.848181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:27.963858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb0a326-a5d2-4a9d-93b5-cc8cdb07004f service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:27.969362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:27.970003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:27.970328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:27.970601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] PATH_INFO: `/v3/users/d3e26ea6703e49b1868e1b4220f4dc95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:27.971898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d3e26ea6703e49b1868e1b4220f4dc95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:27.974103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:28.037166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd3e26ea6703e49b1868e1b4220f4dc95'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.038290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd3e26ea6703e49b1868e1b4220f4dc95'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.049049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd3e26ea6703e49b1868e1b4220f4dc95'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.062037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd3e26ea6703e49b1868e1b4220f4dc95'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.062740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd3e26ea6703e49b1868e1b4220f4dc95'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.078724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 956/1917] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:27 2022] DELETE /identity/v3/users/d3e26ea6703e49b1868e1b4220f4dc95 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:28.083326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:28.083863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:28.083863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:28.104486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:28.105702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:28.167147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 962/1918] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:28.176732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:28.178154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:28.239689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27c910d-6a94-46b6-8a91-b3cac68e2b14 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:28.243731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:28.244336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:28.244613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:28.244864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:28.253923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:28.255378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:28.264797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 957/1919] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:28.545073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:28.547480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:28.572137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25765fea-33b2-4a2f-b83e-62239ba78c0e None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:28.579448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:28.580301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:28.580747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:28.581208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] PATH_INFO: `/v3/projects/9abb9325864746039b46c4859b95a237` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:28.583815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9abb9325864746039b46c4859b95a237)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:28.587777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:28.729167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9abb9325864746039b46c4859b95a237', 'request_id': 'req-294f885b-7761-4461-98ed-6df763322bf0'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.731343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:28.736901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9abb9325864746039b46c4859b95a237', 'request_id': 'req-294f885b-7761-4461-98ed-6df763322bf0'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.747816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9abb9325864746039b46c4859b95a237', 'request_id': 'req-294f885b-7761-4461-98ed-6df763322bf0'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.772279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] The token cache is being invalidate because project 9abb9325864746039b46c4859b95a237 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:28.772574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.774428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.778885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 963/1920] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:28 2022] DELETE /identity/v3/projects/9abb9325864746039b46c4859b95a237 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:28.782901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:28.798606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1bd536-f30b-4520-a4af-a305b0cb6653 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:28.846212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:28.910253 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:28.910784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:28.911095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:28.911371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] PATH_INFO: `/v3/users/ed1ac85416ae4cf6801db1f6e93e138f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:28.920442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed1ac85416ae4cf6801db1f6e93e138f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:28.927128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:28.995709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed1ac85416ae4cf6801db1f6e93e138f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:28.996695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed1ac85416ae4cf6801db1f6e93e138f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.005721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed1ac85416ae4cf6801db1f6e93e138f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.056808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed1ac85416ae4cf6801db1f6e93e138f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.056808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a998bb0b-28a8-40fc-9f9c-44623cd59004 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed1ac85416ae4cf6801db1f6e93e138f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.096585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 958/1921] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:28 2022] DELETE /identity/v3/users/ed1ac85416ae4cf6801db1f6e93e138f => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:29.599908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:29.601361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:29.614700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294f885b-7761-4461-98ed-6df763322bf0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:29.677524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:29.678068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:29.678341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:29.678611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] PATH_INFO: `/v3/projects/6aa33f8b563742109cb2609f66b3945a` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:29.680896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6aa33f8b563742109cb2609f66b3945a)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:29.682427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:29.738548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6aa33f8b563742109cb2609f66b3945a', 'request_id': 'req-8a1be7a6-df34-433c-8921-78f83563ba79'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.745795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6aa33f8b563742109cb2609f66b3945a', 'request_id': 'req-8a1be7a6-df34-433c-8921-78f83563ba79'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.752161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6aa33f8b563742109cb2609f66b3945a', 'request_id': 'req-8a1be7a6-df34-433c-8921-78f83563ba79'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.768698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] The token cache is being invalidate because project 6aa33f8b563742109cb2609f66b3945a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:29.769051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.769801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8a1be7a6-df34-433c-8921-78f83563ba79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:29.772085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 964/1922] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:29 2022] DELETE /identity/v3/projects/6aa33f8b563742109cb2609f66b3945a => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:29.777309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231327ef-1d9f-4563-8afb-bff920a07ad5 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:29.777638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231327ef-1d9f-4563-8afb-bff920a07ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:29.777912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231327ef-1d9f-4563-8afb-bff920a07ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:29.797705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-231327ef-1d9f-4563-8afb-bff920a07ad5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:29.798932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-231327ef-1d9f-4563-8afb-bff920a07ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:29.905557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 959/1923] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:29.913394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae7aa5-f77f-478e-89c8-89df4510e34e None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:29.913679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae7aa5-f77f-478e-89c8-89df4510e34e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:29.914591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae7aa5-f77f-478e-89c8-89df4510e34e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:29.936975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-69ae7aa5-f77f-478e-89c8-89df4510e34e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:29.938536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-69ae7aa5-f77f-478e-89c8-89df4510e34e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:29.952076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 965/1924] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:29.953580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231327ef-1d9f-4563-8afb-bff920a07ad5 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:29.967168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231327ef-1d9f-4563-8afb-bff920a07ad5 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:29.977319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:29.978798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:29.979135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:29.979411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:29.980493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:29.980808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:29.982422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:29.995239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 960/1925] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:29.999487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ae7aa5-f77f-478e-89c8-89df4510e34e None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.009959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ae7aa5-f77f-478e-89c8-89df4510e34e None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.019303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.019924 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.020203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.020524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.021314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.023444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.065503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 966/1926] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:29 2022] POST /identity/v3/projects => generated 385 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:30.070015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.083108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec7a011-e0cf-45f5-8a06-0826218e1ba3 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.118696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:30.150627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.151270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.151550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.151829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.152462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.154392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.174457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 961/1927] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:30 2022] POST /identity/v3/users => generated 293 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:33:30.178575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.189301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307e7036-be8c-48f2-8b15-6dedc68f503a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.193801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.194710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.195052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.195871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.196522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.198126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.212237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 967/1928] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:30.213453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.226084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca2d423-3601-48b8-b36d-f29ee7a2d0c8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.231765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febaf045-422a-4cbc-86d6-7d60ff43f5fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.232461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febaf045-422a-4cbc-86d6-7d60ff43f5fb admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.232739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febaf045-422a-4cbc-86d6-7d60ff43f5fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.233185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febaf045-422a-4cbc-86d6-7d60ff43f5fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.233882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febaf045-422a-4cbc-86d6-7d60ff43f5fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.235614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febaf045-422a-4cbc-86d6-7d60ff43f5fb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.251029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 962/1929] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:30.259877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.286797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60753c38-5d45-496f-94f0-dcca0b3abd9b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.293843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.294131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.294510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.294876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] PATH_INFO: `/v3/projects/c8c80c64deef4c36a6529b3470832af9/users/d101da916cfa48ee9f464c6a8be1408d/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.311374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8c80c64deef4c36a6529b3470832af9, user_id=d101da916cfa48ee9f464c6a8be1408d, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.314570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.329510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 968/1930] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:30 2022] PUT /identity/v3/projects/c8c80c64deef4c36a6529b3470832af9/users/d101da916cfa48ee9f464c6a8be1408d/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:30.346312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34adbc8-9aec-458b-bf22-c79dd5a34a90 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.346598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34adbc8-9aec-458b-bf22-c79dd5a34a90 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.346896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34adbc8-9aec-458b-bf22-c79dd5a34a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.388547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a34adbc8-9aec-458b-bf22-c79dd5a34a90 None None] MFA Rules not processed for user `d101da916cfa48ee9f464c6a8be1408d`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:30.389761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a34adbc8-9aec-458b-bf22-c79dd5a34a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:30.474197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 963/1931] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:30.479115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.489735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:30.562522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a747e69-b96c-4e6e-8490-2e34bba4960a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.567302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.567903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.568183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.568453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.569085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.571883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.611567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 969/1932] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:30 2022] POST /identity/v3/projects => generated 385 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:30.613466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34adbc8-9aec-458b-bf22-c79dd5a34a90 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.625513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34adbc8-9aec-458b-bf22-c79dd5a34a90 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.683375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.684061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.684428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.684765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.685462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.687458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.713827 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 964/1933] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:30 2022] POST /identity/v3/users => generated 299 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:30.717703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.728843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d9e4e-ea6a-48de-9988-c53758181006 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.733728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.734368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.734683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.734982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.735627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.737299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.749159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 970/1934] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:30.753484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.764757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363e749c-25eb-47e9-940b-b4120b04391b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.769316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.769957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.770231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.770500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] PATH_INFO: `/v3/projects/a90fc2555843418eb0a9598a53eddb22/users/2ce173223d5f4270841b883751768cc9/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.786040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a90fc2555843418eb0a9598a53eddb22, user_id=2ce173223d5f4270841b883751768cc9, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.788720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.803124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 965/1935] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:30 2022] PUT /identity/v3/projects/a90fc2555843418eb0a9598a53eddb22/users/2ce173223d5f4270841b883751768cc9/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:30.808065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.821022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959df1d3-c09c-4ae4-8aa7-9eb17893bae9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.877306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2d9a1-9bca-4f5a-ad36-b39fc6fedeaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.877910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2d9a1-9bca-4f5a-ad36-b39fc6fedeaa admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.878229 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2d9a1-9bca-4f5a-ad36-b39fc6fedeaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.878478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2d9a1-9bca-4f5a-ad36-b39fc6fedeaa admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.879156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2d9a1-9bca-4f5a-ad36-b39fc6fedeaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.880751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2d9a1-9bca-4f5a-ad36-b39fc6fedeaa admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.890649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 971/1936] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:30 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:30.894749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:30.901344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:30.906633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a6fa3-3f0d-4a57-99d1-9741f16c0182 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:30.912992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:30.913665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.914422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.914722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] PATH_INFO: `/v3/projects/a90fc2555843418eb0a9598a53eddb22/users/2ce173223d5f4270841b883751768cc9/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.916721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=a90fc2555843418eb0a9598a53eddb22, user_id=2ce173223d5f4270841b883751768cc9, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:30.919387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:30.930275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 966/1937] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:30 2022] PUT /identity/v3/projects/a90fc2555843418eb0a9598a53eddb22/users/2ce173223d5f4270841b883751768cc9/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:30.944781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855d58a5-d6d2-4544-ba67-92c1b71cbf9d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:30.945009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855d58a5-d6d2-4544-ba67-92c1b71cbf9d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:30.945466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855d58a5-d6d2-4544-ba67-92c1b71cbf9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:30.976996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-855d58a5-d6d2-4544-ba67-92c1b71cbf9d None None] MFA Rules not processed for user `2ce173223d5f4270841b883751768cc9`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:30.978929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-855d58a5-d6d2-4544-ba67-92c1b71cbf9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:31.067568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 972/1938] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:30 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:31.080347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:31.082262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:31.156138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618722bf-febb-498b-aff0-fb10c21046ff admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:31.160909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e74d83-b291-43cd-acfa-d4d9a1fc50c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:31.161504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e74d83-b291-43cd-acfa-d4d9a1fc50c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:31.161781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e74d83-b291-43cd-acfa-d4d9a1fc50c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:31.162065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e74d83-b291-43cd-acfa-d4d9a1fc50c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:31.177351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e74d83-b291-43cd-acfa-d4d9a1fc50c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:31.178804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e74d83-b291-43cd-acfa-d4d9a1fc50c3 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:31.189078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 967/1939] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:31 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:31.219091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855d58a5-d6d2-4544-ba67-92c1b71cbf9d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:31.225159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:31.317603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855d58a5-d6d2-4544-ba67-92c1b71cbf9d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:31.322600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:31.323299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:31.323577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:31.323848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] PATH_INFO: `/v3/users/d101da916cfa48ee9f464c6a8be1408d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:31.324981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d101da916cfa48ee9f464c6a8be1408d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:31.327022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:31.392662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd101da916cfa48ee9f464c6a8be1408d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:31.393310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd101da916cfa48ee9f464c6a8be1408d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:31.400738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd101da916cfa48ee9f464c6a8be1408d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:31.401443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd101da916cfa48ee9f464c6a8be1408d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:31.413660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd101da916cfa48ee9f464c6a8be1408d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:31.428138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 973/1940] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:31 2022] DELETE /identity/v3/users/d101da916cfa48ee9f464c6a8be1408d => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:31.432865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:31.433176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:31.433456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:31.453700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:31.455890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:31.520818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 968/1941] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:31.531893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:31.533355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:31.604792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc299a5-baed-4339-9994-63df09897a69 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:31.608541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:31.609225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:31.609592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:31.609924 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:31.620395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:31.622354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:31.633949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 974/1942] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:33:31.968689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:31.970173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:31.982632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee39584-4021-4a93-8e25-090315fca6d7 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:31.987793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:31.988782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:31.989060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:31.989326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] PATH_INFO: `/v3/projects/c8c80c64deef4c36a6529b3470832af9` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:31.991090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8c80c64deef4c36a6529b3470832af9)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:31.993851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:32.041266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8c80c64deef4c36a6529b3470832af9', 'request_id': 'req-831e0885-10a2-4b9d-b2e3-3b47e27a8789'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.047956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8c80c64deef4c36a6529b3470832af9', 'request_id': 'req-831e0885-10a2-4b9d-b2e3-3b47e27a8789'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.053980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8c80c64deef4c36a6529b3470832af9', 'request_id': 'req-831e0885-10a2-4b9d-b2e3-3b47e27a8789'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.076147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] The token cache is being invalidate because project c8c80c64deef4c36a6529b3470832af9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:32.076706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.077500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.079504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 969/1943] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:31 2022] DELETE /identity/v3/projects/c8c80c64deef4c36a6529b3470832af9 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:32.083536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:32.093455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20028527-dcf3-4efc-9ad2-81a338bb6d19 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:32.113826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:32.155762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:32.156393 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:32.156678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:32.156984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] PATH_INFO: `/v3/users/2ce173223d5f4270841b883751768cc9` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:32.158314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ce173223d5f4270841b883751768cc9)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:32.160288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:32.214988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ce173223d5f4270841b883751768cc9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.216059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ce173223d5f4270841b883751768cc9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.227329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ce173223d5f4270841b883751768cc9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.239123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ce173223d5f4270841b883751768cc9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.240812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fb6d22c6-55c7-465a-8bed-e4f2ab066e48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ce173223d5f4270841b883751768cc9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.257359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 975/1944] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:32 2022] DELETE /identity/v3/users/2ce173223d5f4270841b883751768cc9 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:32.600745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:32.603065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:32.618647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831e0885-10a2-4b9d-b2e3-3b47e27a8789 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:32.632389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:32.688607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:32.690341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:32.690341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:32.690341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] PATH_INFO: `/v3/projects/a90fc2555843418eb0a9598a53eddb22` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:32.691233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a90fc2555843418eb0a9598a53eddb22)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:32.693585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:32.757893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a90fc2555843418eb0a9598a53eddb22', 'request_id': 'req-8faac87e-dead-4293-b5f2-d15ec6f475c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.764802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a90fc2555843418eb0a9598a53eddb22', 'request_id': 'req-8faac87e-dead-4293-b5f2-d15ec6f475c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.771105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a90fc2555843418eb0a9598a53eddb22', 'request_id': 'req-8faac87e-dead-4293-b5f2-d15ec6f475c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.787348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] The token cache is being invalidate because project a90fc2555843418eb0a9598a53eddb22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:32.787758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.794331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8faac87e-dead-4293-b5f2-d15ec6f475c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:32.796483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 970/1945] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:32 2022] DELETE /identity/v3/projects/a90fc2555843418eb0a9598a53eddb22 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:32.801317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c4b593-5261-493e-a79d-5b9e97524ce5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:32.802179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c4b593-5261-493e-a79d-5b9e97524ce5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:32.802680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c4b593-5261-493e-a79d-5b9e97524ce5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:32.825008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-24c4b593-5261-493e-a79d-5b9e97524ce5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:32.826482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-24c4b593-5261-493e-a79d-5b9e97524ce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:32.904153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 976/1946] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:32.912421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf3215-2464-4260-837e-9caacfa6e7bb None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:32.912753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf3215-2464-4260-837e-9caacfa6e7bb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:32.912996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cf3215-2464-4260-837e-9caacfa6e7bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:32.934771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-78cf3215-2464-4260-837e-9caacfa6e7bb None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:32.936316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-78cf3215-2464-4260-837e-9caacfa6e7bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:32.945579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 971/1947] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:32.950232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c4b593-5261-493e-a79d-5b9e97524ce5 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:32.962477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c4b593-5261-493e-a79d-5b9e97524ce5 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:32.969394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:32.970087 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:32.970423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:32.970705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:32.971402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:32.971705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:32.973351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:32.988758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 977/1948] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:32.993267 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cf3215-2464-4260-837e-9caacfa6e7bb None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.003139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cf3215-2464-4260-837e-9caacfa6e7bb None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.010509 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.011085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.011370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.011834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.012961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.014990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.048088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 972/1949] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:32 2022] POST /identity/v3/projects => generated 387 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:33.052183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.066233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9967ef0-7d55-4c3f-8534-8cacbdcce9ba admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.097527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:33.129663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.130264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.130546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.130822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.131545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.134445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.160978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 978/1950] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:33 2022] POST /identity/v3/users => generated 294 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:33.162181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.172419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512b2ab1-9b4d-44a8-8053-d564d94588ed admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.177078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.177697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.177943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.178217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.178851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.180501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.196174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 973/1951] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:33:33.202527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.215574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac6e7d6-37a3-4710-b26d-3400348a9284 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.220803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4f8a98-4d5e-4dc8-9d10-1a4acccff887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.221362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f8a98-4d5e-4dc8-9d10-1a4acccff887 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.221676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f8a98-4d5e-4dc8-9d10-1a4acccff887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.221897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f8a98-4d5e-4dc8-9d10-1a4acccff887 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.222563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4f8a98-4d5e-4dc8-9d10-1a4acccff887 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.224216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4f8a98-4d5e-4dc8-9d10-1a4acccff887 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.235899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 979/1952] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:33.240131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.252408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1fd580-e01d-49ee-8c43-4d9dfb832edb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.259511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.260358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.260965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.260965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] PATH_INFO: `/v3/projects/651d67470087400eb709864a072f2f0a/users/90733aab0feb42b78f2a94dc1c37b505/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.280593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] RBAC: Authorizing `identity:create_grant(project_id=651d67470087400eb709864a072f2f0a, user_id=90733aab0feb42b78f2a94dc1c37b505, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.283431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.297160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 974/1953] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:33 2022] PUT /identity/v3/projects/651d67470087400eb709864a072f2f0a/users/90733aab0feb42b78f2a94dc1c37b505/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:33.308235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e8257-aa36-40dd-a174-df282ff77fe2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.308520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e8257-aa36-40dd-a174-df282ff77fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.308791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e8257-aa36-40dd-a174-df282ff77fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.336913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-de9e8257-aa36-40dd-a174-df282ff77fe2 None None] MFA Rules not processed for user `90733aab0feb42b78f2a94dc1c37b505`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:33.338131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-de9e8257-aa36-40dd-a174-df282ff77fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:33.405475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 980/1954] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:33 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:33.409882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.466017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:33.485999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26436fd2-1c12-4ee8-b978-cb95be00c38b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.491268 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.491871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.492306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.492306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.493465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.495407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.522649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 975/1955] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:33 2022] POST /identity/v3/projects => generated 387 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:33.527647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9e8257-aa36-40dd-a174-df282ff77fe2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.552594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9e8257-aa36-40dd-a174-df282ff77fe2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.629044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.629650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.629928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.630197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.630846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.632899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.656925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 981/1956] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:33 2022] POST /identity/v3/users => generated 300 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:33:33.660949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.672663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ada2dd-a7b9-45cb-8e74-9674a5df954d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.677551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.678146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.678425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.678699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.679554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.681257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.695603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 976/1957] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:33.696730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.708159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab3e850-d1e0-4b84-b638-1d879058f6b5 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.714125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.715188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.715534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.715932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] PATH_INFO: `/v3/projects/e8749d8992b24ab7a250c02b8dbfaeae/users/d116ad1001c14d1ebb0d4817a2a4916c/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.749626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8749d8992b24ab7a250c02b8dbfaeae, user_id=d116ad1001c14d1ebb0d4817a2a4916c, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.752333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.766327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 982/1958] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:33 2022] PUT /identity/v3/projects/e8749d8992b24ab7a250c02b8dbfaeae/users/d116ad1001c14d1ebb0d4817a2a4916c/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:33.770320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.781702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90564c9-03da-46ab-889c-7b7d9c10410b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.849505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a893b853-5867-4d19-877d-1388150bef5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.850896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a893b853-5867-4d19-877d-1388150bef5c admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.851201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a893b853-5867-4d19-877d-1388150bef5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.851562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a893b853-5867-4d19-877d-1388150bef5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.852260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a893b853-5867-4d19-877d-1388150bef5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.853886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a893b853-5867-4d19-877d-1388150bef5c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.867374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 977/1959] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:33 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:33.869115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:33.875690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:33.879996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf760fff-e7a1-433d-8e37-8e7a9389dd10 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:33.901399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:33.901399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.901399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.901399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] PATH_INFO: `/v3/projects/e8749d8992b24ab7a250c02b8dbfaeae/users/d116ad1001c14d1ebb0d4817a2a4916c/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.901844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8749d8992b24ab7a250c02b8dbfaeae, user_id=d116ad1001c14d1ebb0d4817a2a4916c, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:33.901844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:33.912745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 983/1960] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:33 2022] PUT /identity/v3/projects/e8749d8992b24ab7a250c02b8dbfaeae/users/d116ad1001c14d1ebb0d4817a2a4916c/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:33.922860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4d795-2a8f-4247-be0c-c1bb5dcc1a85 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:33.923250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4d795-2a8f-4247-be0c-c1bb5dcc1a85 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:33.923585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4d795-2a8f-4247-be0c-c1bb5dcc1a85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:33.952324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0fa4d795-2a8f-4247-be0c-c1bb5dcc1a85 None None] MFA Rules not processed for user `d116ad1001c14d1ebb0d4817a2a4916c`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:33.953547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0fa4d795-2a8f-4247-be0c-c1bb5dcc1a85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:34.024150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 978/1961] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:33 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:34.035023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:34.036759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:34.121557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41a28a0-b2cc-4289-b6cb-1956a07c7b24 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:34.126536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5fc527-7b16-44dc-8921-084a7b062ff8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:34.127179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5fc527-7b16-44dc-8921-084a7b062ff8 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:34.127463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5fc527-7b16-44dc-8921-084a7b062ff8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:34.127735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5fc527-7b16-44dc-8921-084a7b062ff8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:34.137786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5fc527-7b16-44dc-8921-084a7b062ff8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:34.139374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5fc527-7b16-44dc-8921-084a7b062ff8 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:34.150837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 984/1962] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:34 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:34.299842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa4d795-2a8f-4247-be0c-c1bb5dcc1a85 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:34.368810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:34.390325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa4d795-2a8f-4247-be0c-c1bb5dcc1a85 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:34.395427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:34.396210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:34.396488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:34.396759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] PATH_INFO: `/v3/users/90733aab0feb42b78f2a94dc1c37b505` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:34.397887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=90733aab0feb42b78f2a94dc1c37b505)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:34.399904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:34.449848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90733aab0feb42b78f2a94dc1c37b505'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:34.450494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90733aab0feb42b78f2a94dc1c37b505'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:34.459602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90733aab0feb42b78f2a94dc1c37b505'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:34.480008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90733aab0feb42b78f2a94dc1c37b505'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:34.481361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90733aab0feb42b78f2a94dc1c37b505'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:34.495871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 979/1963] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:34 2022] DELETE /identity/v3/users/90733aab0feb42b78f2a94dc1c37b505 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:34.500430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:34.500750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:34.501045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:34.520589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:34.521797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:34.591661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 985/1964] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:34.615941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:34.615941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:34.678833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1b4172-34fa-49fa-b83c-1b0e21099dc9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:34.683150 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:34.683751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:34.684025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:34.684297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:34.693202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:34.694617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:34.704889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 980/1965] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:33:35.042678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:35.044995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:35.062304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa36065-92d3-412d-b199-21ad01298f6b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:35.067160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:35.067972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:35.067972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:35.068247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] PATH_INFO: `/v3/projects/651d67470087400eb709864a072f2f0a` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:35.069428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=651d67470087400eb709864a072f2f0a)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:35.072212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:35.153219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '651d67470087400eb709864a072f2f0a', 'request_id': 'req-dab01700-0f9e-4320-86c8-18ea8d47ae27'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.162693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '651d67470087400eb709864a072f2f0a', 'request_id': 'req-dab01700-0f9e-4320-86c8-18ea8d47ae27'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.170406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '651d67470087400eb709864a072f2f0a', 'request_id': 'req-dab01700-0f9e-4320-86c8-18ea8d47ae27'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.184830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] The token cache is being invalidate because project 651d67470087400eb709864a072f2f0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:35.185136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.185757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.187514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 986/1966] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:35 2022] DELETE /identity/v3/projects/651d67470087400eb709864a072f2f0a => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:35.191232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:35.201494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd030f10-7e94-4c85-a5c3-c56716cb8826 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:35.266945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:35.267601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:35.267878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:35.268181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] PATH_INFO: `/v3/users/d116ad1001c14d1ebb0d4817a2a4916c` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:35.269308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] RBAC: Authorizing `identity:delete_user(user_id=d116ad1001c14d1ebb0d4817a2a4916c)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:35.271255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:35.273846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:35.337820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd116ad1001c14d1ebb0d4817a2a4916c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.338617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd116ad1001c14d1ebb0d4817a2a4916c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.347307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd116ad1001c14d1ebb0d4817a2a4916c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.361924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd116ad1001c14d1ebb0d4817a2a4916c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.362637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4b91a0df-9cfe-4d86-9177-e6d47cb8d8db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd116ad1001c14d1ebb0d4817a2a4916c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.378756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 981/1967] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:35 2022] DELETE /identity/v3/users/d116ad1001c14d1ebb0d4817a2a4916c => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:35.725633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:35.727755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:35.741136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab01700-0f9e-4320-86c8-18ea8d47ae27 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:35.765288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:35.821764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:35.822392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:35.822666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:35.822934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] PATH_INFO: `/v3/projects/e8749d8992b24ab7a250c02b8dbfaeae` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:35.824115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8749d8992b24ab7a250c02b8dbfaeae)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:35.826234 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:35.875403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8749d8992b24ab7a250c02b8dbfaeae', 'request_id': 'req-480e653e-4b7c-4edb-9819-dfb36c77e515'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.882175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8749d8992b24ab7a250c02b8dbfaeae', 'request_id': 'req-480e653e-4b7c-4edb-9819-dfb36c77e515'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.891560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8749d8992b24ab7a250c02b8dbfaeae', 'request_id': 'req-480e653e-4b7c-4edb-9819-dfb36c77e515'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.907985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] The token cache is being invalidate because project e8749d8992b24ab7a250c02b8dbfaeae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:35.908272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.908902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-480e653e-4b7c-4edb-9819-dfb36c77e515 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:35.910419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 987/1968] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:35 2022] DELETE /identity/v3/projects/e8749d8992b24ab7a250c02b8dbfaeae => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:35.916844 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475b011-547e-45de-9383-43f28ee5d0d0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:35.917207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475b011-547e-45de-9383-43f28ee5d0d0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:35.917549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e475b011-547e-45de-9383-43f28ee5d0d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:35.938661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e475b011-547e-45de-9383-43f28ee5d0d0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:35.940087 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e475b011-547e-45de-9383-43f28ee5d0d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:36.014118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 982/1969] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:36.021868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d8d87a-7dc8-4df4-bb19-cf3c4e8e63d6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.022156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d8d87a-7dc8-4df4-bb19-cf3c4e8e63d6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.022431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d8d87a-7dc8-4df4-bb19-cf3c4e8e63d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.057226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-62d8d87a-7dc8-4df4-bb19-cf3c4e8e63d6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:36.059028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-62d8d87a-7dc8-4df4-bb19-cf3c4e8e63d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:36.070796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 988/1970] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:36.074887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475b011-547e-45de-9383-43f28ee5d0d0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.085317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e475b011-547e-45de-9383-43f28ee5d0d0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.090752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.091370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.091648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.093751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.093751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:36.093751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.094597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.109817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 983/1971] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:36.116394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d8d87a-7dc8-4df4-bb19-cf3c4e8e63d6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.128933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d8d87a-7dc8-4df4-bb19-cf3c4e8e63d6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.135960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.136671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.137034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.137377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.138129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.140174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.178991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 989/1972] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:36 2022] POST /identity/v3/projects => generated 365 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:36.183918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.194125 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae84979-4bb1-4a2f-be62-ef60841b8198 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.304937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.305569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.305851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.306126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.307142 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.309204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.327894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:36.343181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.343529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 984/1973] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:36 2022] POST /identity/v3/users => generated 283 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:33:36.357679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17daaab2-7094-4b8b-9a7c-b7267f133349 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.362850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.363503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.363779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.364075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.364711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.366374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.377550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 990/1974] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:36.381447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.391855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6796e404-8e08-4f10-883f-c1f1faad4ea7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.396447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc05981-b958-4227-8744-88abf6ea7d10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.397035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc05981-b958-4227-8744-88abf6ea7d10 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.397309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc05981-b958-4227-8744-88abf6ea7d10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.397577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc05981-b958-4227-8744-88abf6ea7d10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.398247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc05981-b958-4227-8744-88abf6ea7d10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.399851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc05981-b958-4227-8744-88abf6ea7d10 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.410888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 985/1975] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:36.415022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.425032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4e8690-fd55-499b-bd6d-15eb8f61bf63 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.430308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.430991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.431260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.431529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] PATH_INFO: `/v3/projects/28d1052e56e14617b5c2233ddf64fa0f/users/69478be59a024946ab542dd2a273faee/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.451707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=28d1052e56e14617b5c2233ddf64fa0f, user_id=69478be59a024946ab542dd2a273faee, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.454748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.472677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 991/1976] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:36 2022] PUT /identity/v3/projects/28d1052e56e14617b5c2233ddf64fa0f/users/69478be59a024946ab542dd2a273faee/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:36.484352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddce11b8-ce88-40c2-8423-1b9864307c2f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.484639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddce11b8-ce88-40c2-8423-1b9864307c2f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.484911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddce11b8-ce88-40c2-8423-1b9864307c2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.515883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ddce11b8-ce88-40c2-8423-1b9864307c2f None None] MFA Rules not processed for user `69478be59a024946ab542dd2a273faee`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:36.517154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ddce11b8-ce88-40c2-8423-1b9864307c2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:36.595846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 986/1977] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:36 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:36.602216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.654078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:36.674090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797730a6-ee11-4954-8efd-13cd5219fd2d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.678718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.679439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.679721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.679986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.680701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.682624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.724546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 992/1978] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:36 2022] POST /identity/v3/projects => generated 365 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:36.728967 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddce11b8-ce88-40c2-8423-1b9864307c2f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.741052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddce11b8-ce88-40c2-8423-1b9864307c2f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.800346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.800946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.801218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.801486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.802136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.804058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.833492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 987/1979] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:36 2022] POST /identity/v3/users => generated 289 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:36.834711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.848536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b8de9f-c0c9-4744-9c84-c00b40379ca4 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.853505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.854103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.854380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.854648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.855324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.856915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.868744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 993/1980] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:36.872954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.884381 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b819c8-a979-4aaf-bae1-16e082909c02 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:36.889638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:36.890290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:36.890568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:36.890842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] PATH_INFO: `/v3/projects/ceea8e2dc0cb448083e8920bc3444210/users/9b609f03aa4c4892a08f1745912d1cfb/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:36.907197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] RBAC: Authorizing `identity:create_grant(project_id=ceea8e2dc0cb448083e8920bc3444210, user_id=9b609f03aa4c4892a08f1745912d1cfb, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:36.910516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:36.932168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 988/1981] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:36 2022] PUT /identity/v3/projects/ceea8e2dc0cb448083e8920bc3444210/users/9b609f03aa4c4892a08f1745912d1cfb/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:36.936449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:36.946327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea50d8-a288-40ec-916a-3adb24849670 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:37.006516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799ba9ce-3ba1-4d56-88a7-0357f8a1adde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:37.007160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ba9ce-3ba1-4d56-88a7-0357f8a1adde admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:37.007497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ba9ce-3ba1-4d56-88a7-0357f8a1adde admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:37.008045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ba9ce-3ba1-4d56-88a7-0357f8a1adde admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:37.008923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799ba9ce-3ba1-4d56-88a7-0357f8a1adde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:37.010581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799ba9ce-3ba1-4d56-88a7-0357f8a1adde admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:37.021038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 994/1982] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:36 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:37.025074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:37.035925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112be629-5d0a-4d2e-b7e5-636a188c5adf admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:37.041215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:37.041868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:37.042143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:37.042443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] PATH_INFO: `/v3/projects/ceea8e2dc0cb448083e8920bc3444210/users/9b609f03aa4c4892a08f1745912d1cfb/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:37.044493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ceea8e2dc0cb448083e8920bc3444210, user_id=9b609f03aa4c4892a08f1745912d1cfb, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:37.047149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:37.059858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 989/1983] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:37 2022] PUT /identity/v3/projects/ceea8e2dc0cb448083e8920bc3444210/users/9b609f03aa4c4892a08f1745912d1cfb/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:37.070896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:37.071451 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:37.071646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:37.106293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] MFA Rules not processed for user `9b609f03aa4c4892a08f1745912d1cfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:37.107542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:37.188250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 995/1984] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:37 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:37.199530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:37.203187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:37.238036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:37.266899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653e0ba2-46e4-4581-b084-436a55f46c3c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:37.271222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7975c5d3-e453-48a0-ab09-31b8199a4bef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:37.271829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7975c5d3-e453-48a0-ab09-31b8199a4bef service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:37.272100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7975c5d3-e453-48a0-ab09-31b8199a4bef service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:37.272367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7975c5d3-e453-48a0-ab09-31b8199a4bef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:37.281941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7975c5d3-e453-48a0-ab09-31b8199a4bef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:37.283380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7975c5d3-e453-48a0-ab09-31b8199a4bef service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:37.292898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 990/1985] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:37 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:37.402799 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:37.407013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:37.439096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80806f13-8145-4de8-9cad-c2c676bae7cb None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:37.444834 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd38d886-d3cf-46c7-abe3-bb87e9e36cea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:37.448066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd38d886-d3cf-46c7-abe3-bb87e9e36cea service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:37.448389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd38d886-d3cf-46c7-abe3-bb87e9e36cea service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:37.448646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd38d886-d3cf-46c7-abe3-bb87e9e36cea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:37.508567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:37.528901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd38d886-d3cf-46c7-abe3-bb87e9e36cea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:37.531262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd38d886-d3cf-46c7-abe3-bb87e9e36cea service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:37.540794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 996/1986] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:37 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:38.563723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63acf50-092c-4b09-845a-902c0a71e756 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:38.564588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63acf50-092c-4b09-845a-902c0a71e756 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:38.565026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63acf50-092c-4b09-845a-902c0a71e756 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:38.567052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 991/1987] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:33:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:33:38.583666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad9d0bf-e330-4e5c-ad9d-d99c7dc8d9a5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:38.584186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad9d0bf-e330-4e5c-ad9d-d99c7dc8d9a5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:38.584266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad9d0bf-e330-4e5c-ad9d-d99c7dc8d9a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:38.609484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-8ad9d0bf-e330-4e5c-ad9d-d99c7dc8d9a5 None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:38.611144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8ad9d0bf-e330-4e5c-ad9d-d99c7dc8d9a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:38.696163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 997/1988] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:33:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:38.728290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63acf50-092c-4b09-845a-902c0a71e756 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:38.730724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b63acf50-092c-4b09-845a-902c0a71e756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:38.772517 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63acf50-092c-4b09-845a-902c0a71e756 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:38.786129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bc12c0-0b85-4d12-8b0d-7dd81901b4ca service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:38.786879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bc12c0-0b85-4d12-8b0d-7dd81901b4ca service placement] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:38.787872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bc12c0-0b85-4d12-8b0d-7dd81901b4ca service placement] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:38.789474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bc12c0-0b85-4d12-8b0d-7dd81901b4ca service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:38.819084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50bc12c0-0b85-4d12-8b0d-7dd81901b4ca service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:38.821168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50bc12c0-0b85-4d12-8b0d-7dd81901b4ca service placement] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:38.836055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 992/1989] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:39.389760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70384cb-6f28-4b99-b600-20a0b104be96 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:39.390208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70384cb-6f28-4b99-b600-20a0b104be96 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:39.390555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70384cb-6f28-4b99-b600-20a0b104be96 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:39.391999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 998/1990] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:33:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:33:39.410729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:39.411187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:39.411469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:39.433978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:39.435243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:39.444044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 993/1991] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:33:39 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:39.459612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70384cb-6f28-4b99-b600-20a0b104be96 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:39.475905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70384cb-6f28-4b99-b600-20a0b104be96 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:39.481966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:39.482573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:39.483284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:39.483284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:39.492582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:39.499022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:39.503776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:39.505481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:39.511135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 999/1992] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:39.576594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31ffdcd-c2b6-488b-90d5-8416cf102fdf None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:39.581096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:39.581713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:39.581987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:39.582255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] PATH_INFO: `/v3/users/e4598fe8b3364ca0b3a818edd0a54ff2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:39.583394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4598fe8b3364ca0b3a818edd0a54ff2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:39.585284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:39.629756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4598fe8b3364ca0b3a818edd0a54ff2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:39.630390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4598fe8b3364ca0b3a818edd0a54ff2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:39.637432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4598fe8b3364ca0b3a818edd0a54ff2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:39.650358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4598fe8b3364ca0b3a818edd0a54ff2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:39.651095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4598fe8b3364ca0b3a818edd0a54ff2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:39.665383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 994/1993] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:39 2022] DELETE /identity/v3/users/e4598fe8b3364ca0b3a818edd0a54ff2 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:39.904517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:39.918169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2085470e-c2de-4c56-ae2d-74eb98f164f5 service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:40.007351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:40.007980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:40.008254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:40.008523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] PATH_INFO: `/v3/projects/a5333719f29749769b2f1052e453d4dd` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:40.009680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5333719f29749769b2f1052e453d4dd)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:40.011581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:40.066299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5333719f29749769b2f1052e453d4dd', 'request_id': 'req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.094582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5333719f29749769b2f1052e453d4dd', 'request_id': 'req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.116127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5333719f29749769b2f1052e453d4dd', 'request_id': 'req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.119191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:40.131915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] The token cache is being invalidate because project a5333719f29749769b2f1052e453d4dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:40.132418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.133262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.134785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1000/1994] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:39 2022] DELETE /identity/v3/projects/a5333719f29749769b2f1052e453d4dd => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:40.138618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:40.140022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:40.153795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb3f5cc-3eb6-47fa-977c-30b71332bba0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:40.184788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:40.221948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:40.222608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:40.222889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:40.223195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] PATH_INFO: `/v3/users/b85310a65d03474c93225c8d31e6a730` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:40.227690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b85310a65d03474c93225c8d31e6a730)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:40.229693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:40.278426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b85310a65d03474c93225c8d31e6a730'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.279338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b85310a65d03474c93225c8d31e6a730'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.286141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b85310a65d03474c93225c8d31e6a730'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.298205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b85310a65d03474c93225c8d31e6a730'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.298897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-da8c3d12-794d-4496-ac84-0d5a65b42d8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b85310a65d03474c93225c8d31e6a730'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.313842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 995/1995] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:40 2022] DELETE /identity/v3/users/b85310a65d03474c93225c8d31e6a730 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:40.522771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:40.525177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:40.542496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ca10f9-8ddc-43d3-af1d-b7c088a50fa3 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:40.608955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:40.609821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:40.610325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:40.610884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] PATH_INFO: `/v3/projects/531d455406924a40b95f3de63b192e9a` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:40.612749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] RBAC: Authorizing `identity:delete_project(project_id=531d455406924a40b95f3de63b192e9a)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:40.615057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:40.685673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '531d455406924a40b95f3de63b192e9a', 'request_id': 'req-522c1972-c220-4a59-9c77-44b4dacd4276'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.692238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '531d455406924a40b95f3de63b192e9a', 'request_id': 'req-522c1972-c220-4a59-9c77-44b4dacd4276'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.700255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '531d455406924a40b95f3de63b192e9a', 'request_id': 'req-522c1972-c220-4a59-9c77-44b4dacd4276'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.733781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] The token cache is being invalidate because project 531d455406924a40b95f3de63b192e9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:40.734082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.734829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-522c1972-c220-4a59-9c77-44b4dacd4276 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:40.740933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1001/1996] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:40 2022] DELETE /identity/v3/projects/531d455406924a40b95f3de63b192e9a => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:40.743048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ddff8c-b755-4715-91cf-014e47c73908 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:40.743508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ddff8c-b755-4715-91cf-014e47c73908 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:40.743749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ddff8c-b755-4715-91cf-014e47c73908 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:40.772229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e7ddff8c-b755-4715-91cf-014e47c73908 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:40.773991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e7ddff8c-b755-4715-91cf-014e47c73908 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:40.877104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 996/1997] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:40.886044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ac475-d0d1-4645-bc82-5a124f95c136 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:40.886342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ac475-d0d1-4645-bc82-5a124f95c136 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:40.886950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ac475-d0d1-4645-bc82-5a124f95c136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:40.909694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d44ac475-d0d1-4645-bc82-5a124f95c136 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:40.912372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d44ac475-d0d1-4645-bc82-5a124f95c136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:40.924293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1002/1998] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:40.931576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ddff8c-b755-4715-91cf-014e47c73908 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:40.942769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ddff8c-b755-4715-91cf-014e47c73908 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:40.947733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:40.948342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:40.948759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:40.949038 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:40.949687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:40.950114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:40.951917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:40.967081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 997/1999] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:33:40.972047 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ac475-d0d1-4645-bc82-5a124f95c136 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:40.985203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44ac475-d0d1-4645-bc82-5a124f95c136 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:40.991443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:40.993475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:40.993946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:40.994871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:40.995467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:40.997520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.037127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1003/2000] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:40 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:41.042478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.057700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c6023d-69c3-42a8-8cf8-378510319ea0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.109208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:41.154562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.155240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.155522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.155801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.156445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.158551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.199819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 998/2001] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:41 2022] POST /identity/v3/users => generated 288 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:41.203575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.220648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c967acf9-f795-4a22-b58f-8e1f63e5b28a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.232681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.233335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.233683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.234026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.234751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.236840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.280422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1004/2002] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:41 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:41.280790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.315652 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf1142-1a0f-4318-86d5-efe88a109438 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.324103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b55a1e-3c83-4e85-8329-d9d2c2e65e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.325500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b55a1e-3c83-4e85-8329-d9d2c2e65e12 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.325895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b55a1e-3c83-4e85-8329-d9d2c2e65e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.326367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b55a1e-3c83-4e85-8329-d9d2c2e65e12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.327121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b55a1e-3c83-4e85-8329-d9d2c2e65e12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.329225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b55a1e-3c83-4e85-8329-d9d2c2e65e12 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.354280 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.360963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 999/2003] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:41 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:41.370048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367aedcc-2193-43bd-8ebc-f16da75b7e01 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.375993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.376667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.376935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.377210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] PATH_INFO: `/v3/projects/dd87fe693f8f4e7184d2d05992bbc9b2/users/549915f363f24d71bab9a899fa664d32/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.414078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd87fe693f8f4e7184d2d05992bbc9b2, user_id=549915f363f24d71bab9a899fa664d32, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.417845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.441042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1005/2004] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:41 2022] PUT /identity/v3/projects/dd87fe693f8f4e7184d2d05992bbc9b2/users/549915f363f24d71bab9a899fa664d32/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:41.524925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebd6901-91b5-4da3-802b-fb65a5513664 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.525215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebd6901-91b5-4da3-802b-fb65a5513664 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.525499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebd6901-91b5-4da3-802b-fb65a5513664 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.562030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2ebd6901-91b5-4da3-802b-fb65a5513664 None None] MFA Rules not processed for user `549915f363f24d71bab9a899fa664d32`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:41.563343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2ebd6901-91b5-4da3-802b-fb65a5513664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:41.637271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1000/2005] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:41.641603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.663467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:INSERT=4,keystone:DELETE=8,keystone:SELECT=88 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:41.715162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f71164-5843-4018-a66c-4723d03dc382 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.719896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.720504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.720778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.721067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.721822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.723821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.761095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1006/2006] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:41 2022] POST /identity/v3/projects => generated 373 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:41.766187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebd6901-91b5-4da3-802b-fb65a5513664 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.780949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebd6901-91b5-4da3-802b-fb65a5513664 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.846376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.847015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.847290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.847723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.848515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.850551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.876655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1001/2007] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:41 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:41.877951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.890466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04f4a44-5573-4fa2-aa9e-e6990acc91a7 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.895354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.895962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.896243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.896516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.897160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.899879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.910990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1007/2008] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:41.916143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.927513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a6064-2647-4bda-a700-28639741b868 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:41.933167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:41.933820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:41.934100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:41.934372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] PATH_INFO: `/v3/projects/ee6f047c2fe244f1857c7b201369c647/users/5bcdcfb91bd54680b710733c43d827b7/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:41.950422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee6f047c2fe244f1857c7b201369c647, user_id=5bcdcfb91bd54680b710733c43d827b7, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:41.953110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:41.968984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1002/2009] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:41 2022] PUT /identity/v3/projects/ee6f047c2fe244f1857c7b201369c647/users/5bcdcfb91bd54680b710733c43d827b7/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:41.973111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:41.985665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbeea06-9063-4b2e-b575-9c0af6126408 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:42.052463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36678a1d-b284-4bdd-97ef-07feac7f216e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:42.053016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36678a1d-b284-4bdd-97ef-07feac7f216e admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:42.053295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36678a1d-b284-4bdd-97ef-07feac7f216e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:42.053572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36678a1d-b284-4bdd-97ef-07feac7f216e admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:42.054362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36678a1d-b284-4bdd-97ef-07feac7f216e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:42.056682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36678a1d-b284-4bdd-97ef-07feac7f216e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:42.067681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1008/2010] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:41 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:42.072760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:42.083457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e2200-6d94-43e1-a3b4-b03267330b3a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:42.089242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:42.089896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:42.090170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:42.090439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] PATH_INFO: `/v3/projects/ee6f047c2fe244f1857c7b201369c647/users/5bcdcfb91bd54680b710733c43d827b7/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:42.092584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee6f047c2fe244f1857c7b201369c647, user_id=5bcdcfb91bd54680b710733c43d827b7, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:42.095349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:42.101790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:42.109476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1003/2011] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:42 2022] PUT /identity/v3/projects/ee6f047c2fe244f1857c7b201369c647/users/5bcdcfb91bd54680b710733c43d827b7/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:42.122637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:42.122920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:42.123261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:42.153336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] MFA Rules not processed for user `5bcdcfb91bd54680b710733c43d827b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:42.154567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:42.246719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1009/2012] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:42 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:42.257457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:42.260936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:42.327078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933fd882-4521-4d8c-ad58-c21cd9aaaa5f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:42.331006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97ccf9a-a28b-4c1f-af3a-2d469f67bcd9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:42.331513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ccf9a-a28b-4c1f-af3a-2d469f67bcd9 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:42.331751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ccf9a-a28b-4c1f-af3a-2d469f67bcd9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:42.332000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ccf9a-a28b-4c1f-af3a-2d469f67bcd9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:42.394107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97ccf9a-a28b-4c1f-af3a-2d469f67bcd9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:42.395587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97ccf9a-a28b-4c1f-af3a-2d469f67bcd9 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:42.406504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1004/2013] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:43.065410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:43.067323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:43.084744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:43.151417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c42479f-34f9-43a4-9a07-c56bba83e14b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:43.156937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:43.157695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:43.158041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:43.158386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] PATH_INFO: `/v3/users/69478be59a024946ab542dd2a273faee` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:43.160446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=69478be59a024946ab542dd2a273faee)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:43.162532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:43.227567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69478be59a024946ab542dd2a273faee'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.228597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69478be59a024946ab542dd2a273faee'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.236260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69478be59a024946ab542dd2a273faee'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.236923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69478be59a024946ab542dd2a273faee'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.249565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69478be59a024946ab542dd2a273faee'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.263381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1010/2014] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:43 2022] DELETE /identity/v3/users/69478be59a024946ab542dd2a273faee => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:43.268025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:43.268622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:43.268622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:43.287805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:43.289278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:43.357743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1005/2015] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:43.373168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:43.376893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:43.442104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c52fabc-da6c-41e1-ab0d-3edb0c0b5e5e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:43.446025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:43.446632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:43.446909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:43.447210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:43.456361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:43.457791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:43.468407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1011/2016] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:43.839103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:43.840764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:43.860459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3be431-8501-44cc-9c95-4de1b6f554d1 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:43.868050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:43.868519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:43.868853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:43.869789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] PATH_INFO: `/v3/projects/28d1052e56e14617b5c2233ddf64fa0f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:43.872454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28d1052e56e14617b5c2233ddf64fa0f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:43.875849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:43.937500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:43.941006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28d1052e56e14617b5c2233ddf64fa0f', 'request_id': 'req-71fea292-46f5-441d-b722-ef68d1c83572'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.951042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28d1052e56e14617b5c2233ddf64fa0f', 'request_id': 'req-71fea292-46f5-441d-b722-ef68d1c83572'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.957140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28d1052e56e14617b5c2233ddf64fa0f', 'request_id': 'req-71fea292-46f5-441d-b722-ef68d1c83572'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.975214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] The token cache is being invalidate because project 28d1052e56e14617b5c2233ddf64fa0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:43.975635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.976684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:43.983263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1006/2017] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:43 2022] DELETE /identity/v3/projects/28d1052e56e14617b5c2233ddf64fa0f => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:43.984914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:43.996982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efd3772-0fe2-4c89-8e24-a68a337e5b95 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:44.033580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:44.068968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:44.070327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:44.070669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:44.071096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] PATH_INFO: `/v3/users/9b609f03aa4c4892a08f1745912d1cfb` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:44.072701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b609f03aa4c4892a08f1745912d1cfb)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:44.074368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:44.138344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b609f03aa4c4892a08f1745912d1cfb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.139045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b609f03aa4c4892a08f1745912d1cfb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.147039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b609f03aa4c4892a08f1745912d1cfb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.147689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b609f03aa4c4892a08f1745912d1cfb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.160426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-99e2ffc6-12c2-40f7-8f27-0f68647abafd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b609f03aa4c4892a08f1745912d1cfb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.173606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1012/2018] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:43 2022] DELETE /identity/v3/users/9b609f03aa4c4892a08f1745912d1cfb => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:44.503946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:44.506153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:44.538056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fea292-46f5-441d-b722-ef68d1c83572 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:44.614258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:44.615022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:44.615374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:44.615711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] PATH_INFO: `/v3/projects/ceea8e2dc0cb448083e8920bc3444210` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:44.618591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ceea8e2dc0cb448083e8920bc3444210)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:44.621258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:44.678389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ceea8e2dc0cb448083e8920bc3444210', 'request_id': 'req-19090e13-d9d3-4667-b647-836a41f61ca9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.688004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ceea8e2dc0cb448083e8920bc3444210', 'request_id': 'req-19090e13-d9d3-4667-b647-836a41f61ca9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.698765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ceea8e2dc0cb448083e8920bc3444210', 'request_id': 'req-19090e13-d9d3-4667-b647-836a41f61ca9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.712910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] The token cache is being invalidate because project ceea8e2dc0cb448083e8920bc3444210 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:44.713201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.713818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-19090e13-d9d3-4667-b647-836a41f61ca9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:44.715411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1007/2019] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:44 2022] DELETE /identity/v3/projects/ceea8e2dc0cb448083e8920bc3444210 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:44.720456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec295c42-3a24-4f46-b15f-8dc97dc4cb3f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:44.720734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec295c42-3a24-4f46-b15f-8dc97dc4cb3f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:44.721029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec295c42-3a24-4f46-b15f-8dc97dc4cb3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:44.741434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ec295c42-3a24-4f46-b15f-8dc97dc4cb3f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:44.744009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec295c42-3a24-4f46-b15f-8dc97dc4cb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:44.808987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1013/2020] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:44.819402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70eb7f-b04e-4576-b299-ae2b6ae91677 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:44.819766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70eb7f-b04e-4576-b299-ae2b6ae91677 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:44.821279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70eb7f-b04e-4576-b299-ae2b6ae91677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:44.842715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ed70eb7f-b04e-4576-b299-ae2b6ae91677 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:44.843968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ed70eb7f-b04e-4576-b299-ae2b6ae91677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:44.853324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1008/2021] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:44.857189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec295c42-3a24-4f46-b15f-8dc97dc4cb3f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:44.868892 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec295c42-3a24-4f46-b15f-8dc97dc4cb3f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:44.874030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:44.874765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:44.875079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:44.875349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:44.876001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:33:44.876299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:44.878578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:44.892524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1014/2022] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:33:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:33:44.896726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70eb7f-b04e-4576-b299-ae2b6ae91677 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:44.909595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70eb7f-b04e-4576-b299-ae2b6ae91677 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:44.914931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:44.915695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:44.916096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:44.916429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:44.917073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:44.919810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:44.956808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1009/2023] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:33:44 2022] POST /identity/v3/projects => generated 373 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:33:44.958057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:44.970227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b013260b-d053-4db7-b7a4-43bd966a0727 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:45.015255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:45.037407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:45.038177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.039117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.039525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.040173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:45.042241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:45.067033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1015/2024] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:33:44 2022] POST /identity/v3/users => generated 287 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:33:45.071135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:45.082023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9bcc7-a0ef-4b37-ab9f-79143672f8af admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:45.087853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:45.088458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.088733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.088995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.089630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:45.091224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:45.102070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1010/2025] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:45.106239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:45.116879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6718b719-ccbe-42d4-b59f-3dac15c9a2b8 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:45.121844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:45.122448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.122857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.123188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.123828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:45.125408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:45.137188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1016/2026] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:33:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:33:45.141108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:45.151996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c380783-6bf0-4fa9-acc0-e8ec2358af9a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:45.152557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:45.154246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:45.157126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89897c3c-89ce-4190-a470-4175055ec5a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:45.157791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89897c3c-89ce-4190-a470-4175055ec5a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.158093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89897c3c-89ce-4190-a470-4175055ec5a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.158335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89897c3c-89ce-4190-a470-4175055ec5a9 admin admin] PATH_INFO: `/v3/projects/250b86943e2d45dc992421c2adee42b2/users/db187750f7514609923133b9a5f17ae7/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.168193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850f501e-cc4c-4e5a-a7a9-f4b2ac93571f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:45.174257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:45.174891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.175192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.175463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] PATH_INFO: `/v3/users/549915f363f24d71bab9a899fa664d32` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.176580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] RBAC: Authorizing `identity:delete_user(user_id=549915f363f24d71bab9a899fa664d32)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:45.178507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:45.179134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89897c3c-89ce-4190-a470-4175055ec5a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=250b86943e2d45dc992421c2adee42b2, user_id=db187750f7514609923133b9a5f17ae7, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:45.182544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89897c3c-89ce-4190-a470-4175055ec5a9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:45.200733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1011/2027] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:33:45 2022] PUT /identity/v3/projects/250b86943e2d45dc992421c2adee42b2/users/db187750f7514609923133b9a5f17ae7/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:45.205698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.205871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.206086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.213616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:45.230225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:45.231469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:45.237628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '549915f363f24d71bab9a899fa664d32'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:45.238415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '549915f363f24d71bab9a899fa664d32'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:45.246965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '549915f363f24d71bab9a899fa664d32'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:45.247787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '549915f363f24d71bab9a899fa664d32'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:45.269207 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c9615cc8-e33c-4c94-9cff-a989bbf4f135 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '549915f363f24d71bab9a899fa664d32'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:45.286642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1017/2028] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:45 2022] DELETE /identity/v3/users/549915f363f24d71bab9a899fa664d32 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:45.292064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.292530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.292945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.302380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1012/2029] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:45.320515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:45.322698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:45.322824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:45.324082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:45.389297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8481e98c-556b-4d7a-a6ed-f2de793eeeb2 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:45.448466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1018/2030] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:45.464484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:45.464707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:45.465044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.465515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.465788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.466230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:45.479042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:45.482508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:45.501569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1013/2031] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:45.506566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51d469b-c31a-4eff-b865-a6b9d7ec471e None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:45.517005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:45.521439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:45.526151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:45.526151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:45.547956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:45.549936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:45.569075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1019/2032] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:33:45.988372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:45.990295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:46.029260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaefb6e-9edf-4fb0-928a-3604ae158c38 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:46.043236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:46.044637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:46.045105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:46.045377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] PATH_INFO: `/v3/projects/dd87fe693f8f4e7184d2d05992bbc9b2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:46.046555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd87fe693f8f4e7184d2d05992bbc9b2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:46.048545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:46.181946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd87fe693f8f4e7184d2d05992bbc9b2', 'request_id': 'req-96af5620-4df9-4da6-ae85-6ddd1e0ec350'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.189579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd87fe693f8f4e7184d2d05992bbc9b2', 'request_id': 'req-96af5620-4df9-4da6-ae85-6ddd1e0ec350'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.192809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:46.199986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd87fe693f8f4e7184d2d05992bbc9b2', 'request_id': 'req-96af5620-4df9-4da6-ae85-6ddd1e0ec350'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.221356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] The token cache is being invalidate because project dd87fe693f8f4e7184d2d05992bbc9b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:46.223846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.224990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-96af5620-4df9-4da6-ae85-6ddd1e0ec350 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.227780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1014/2033] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:45 2022] DELETE /identity/v3/projects/dd87fe693f8f4e7184d2d05992bbc9b2 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:46.231755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:46.249279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b2f948-bef1-41b7-b749-74ddd7add548 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:46.277226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:46.342859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:46.343879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:46.344382 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:46.345564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] PATH_INFO: `/v3/users/5bcdcfb91bd54680b710733c43d827b7` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:46.347092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bcdcfb91bd54680b710733c43d827b7)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:46.349211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:46.406003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bcdcfb91bd54680b710733c43d827b7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.406657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bcdcfb91bd54680b710733c43d827b7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.414161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bcdcfb91bd54680b710733c43d827b7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.415370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bcdcfb91bd54680b710733c43d827b7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.430883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bcdcfb91bd54680b710733c43d827b7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:46.458287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1020/2034] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:46 2022] DELETE /identity/v3/users/5bcdcfb91bd54680b710733c43d827b7 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:33:46.694211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:46.694547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:46.695577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:46.724633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] MFA Rules not processed for user `db187750f7514609923133b9a5f17ae7`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:46.726478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:46.805293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:46.806779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:46.816855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1015/2035] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:46.828821 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:46.830375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:46.908261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db27ad6-7fa2-41cb-b08c-5749780b3309 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:46.913158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c7966b-09e1-48f6-81b8-f5ffc6e92feb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:46.914021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7966b-09e1-48f6-81b8-f5ffc6e92feb service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:46.914390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7966b-09e1-48f6-81b8-f5ffc6e92feb service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:46.914786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c7966b-09e1-48f6-81b8-f5ffc6e92feb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:46.917243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea4825d-4834-4984-bdac-5b293c4649d1 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:46.922379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:46.923912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:46.924347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:46.924768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] PATH_INFO: `/v3/projects/ee6f047c2fe244f1857c7b201369c647` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:46.926238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c7966b-09e1-48f6-81b8-f5ffc6e92feb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:46.926664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee6f047c2fe244f1857c7b201369c647)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:46.928002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c7966b-09e1-48f6-81b8-f5ffc6e92feb service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:46.929534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:46.956422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1016/2036] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:33:46.998479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:47.018122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee6f047c2fe244f1857c7b201369c647', 'request_id': 'req-8e26076d-33b7-4af9-b40f-45669fc6640d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:47.034927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee6f047c2fe244f1857c7b201369c647', 'request_id': 'req-8e26076d-33b7-4af9-b40f-45669fc6640d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:47.051710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee6f047c2fe244f1857c7b201369c647', 'request_id': 'req-8e26076d-33b7-4af9-b40f-45669fc6640d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:47.088882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] The token cache is being invalidate because project ee6f047c2fe244f1857c7b201369c647 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:33:47.089216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:47.095161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:47.099361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1021/2037] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:33:46 2022] DELETE /identity/v3/projects/ee6f047c2fe244f1857c7b201369c647 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:48.674611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2e324a-c4a4-4da3-bbc6-9264fc2e48c0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:48.675177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2e324a-c4a4-4da3-bbc6-9264fc2e48c0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:48.675257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2e324a-c4a4-4da3-bbc6-9264fc2e48c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:48.704786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7b2e324a-c4a4-4da3-bbc6-9264fc2e48c0 None None] MFA Rules not processed for user `db187750f7514609923133b9a5f17ae7`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:48.706014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7b2e324a-c4a4-4da3-bbc6-9264fc2e48c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:48.811373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1017/2038] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:33:57.077097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=9,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:58.795054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=87,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:33:59.336583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:59.338079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:59.411012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e26076d-33b7-4af9-b40f-45669fc6640d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:59.415771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:59.416576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:59.416956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:59.417326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] PATH_INFO: `/v3/users/baa7f562ab1b401faa4796135e4b1273` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:59.418595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] RBAC: Authorizing `identity:delete_user(user_id=baa7f562ab1b401faa4796135e4b1273)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:59.420702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:59.476814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baa7f562ab1b401faa4796135e4b1273'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:59.477850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baa7f562ab1b401faa4796135e4b1273'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:59.486610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baa7f562ab1b401faa4796135e4b1273'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:59.487431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baa7f562ab1b401faa4796135e4b1273'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:59.503854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baa7f562ab1b401faa4796135e4b1273'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:33:59.523046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1022/2039] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:33:59 2022] DELETE /identity/v3/users/baa7f562ab1b401faa4796135e4b1273 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:33:59.525597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:59.525892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:59.526164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:59.547276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:33:59.548843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:59.635114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1018/2040] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:33:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:33:59.645170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:33:59.646014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:33:59.711787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9a9372-2d8b-4da0-bbdb-90ea0e598581 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:33:59.716356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee5fcea-490a-4b4e-baae-d9b2a0e1b97e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:33:59.716984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee5fcea-490a-4b4e-baae-d9b2a0e1b97e service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:33:59.717260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee5fcea-490a-4b4e-baae-d9b2a0e1b97e service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:33:59.717529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee5fcea-490a-4b4e-baae-d9b2a0e1b97e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:33:59.727806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee5fcea-490a-4b4e-baae-d9b2a0e1b97e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:33:59.729262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee5fcea-490a-4b4e-baae-d9b2a0e1b97e service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:33:59.744682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1023/2041] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:33:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:00.172473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:00.174169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:00.197657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfe8031-4321-4033-ae2d-e70b11e0f566 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:00.203960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:00.204910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.205400 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.205878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] PATH_INFO: `/v3/projects/0a93d5a70da4473089e32d973268d35f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.207391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a93d5a70da4473089e32d973268d35f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:00.220294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:00.325724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a93d5a70da4473089e32d973268d35f', 'request_id': 'req-be66fc2a-6d3e-4a1d-a37b-93389227958e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:00.343316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a93d5a70da4473089e32d973268d35f', 'request_id': 'req-be66fc2a-6d3e-4a1d-a37b-93389227958e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:00.352114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a93d5a70da4473089e32d973268d35f', 'request_id': 'req-be66fc2a-6d3e-4a1d-a37b-93389227958e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:00.373788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] The token cache is being invalidate because project 0a93d5a70da4473089e32d973268d35f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:00.374083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:00.374775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-be66fc2a-6d3e-4a1d-a37b-93389227958e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:00.376970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1019/2042] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:00 2022] DELETE /identity/v3/projects/0a93d5a70da4473089e32d973268d35f => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:34:00.383434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca216e8-372d-4cad-840d-1375085076e6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.383955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca216e8-372d-4cad-840d-1375085076e6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.384053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca216e8-372d-4cad-840d-1375085076e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.404296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-eca216e8-372d-4cad-840d-1375085076e6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:00.405510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-eca216e8-372d-4cad-840d-1375085076e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:00.427455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:00.480181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1024/2043] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:00.488058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed300c74-3b70-4f03-853c-fa61b4a7f1df None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.488343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed300c74-3b70-4f03-853c-fa61b4a7f1df None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.489121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed300c74-3b70-4f03-853c-fa61b4a7f1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.509945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ed300c74-3b70-4f03-853c-fa61b4a7f1df None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:00.511174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ed300c74-3b70-4f03-853c-fa61b4a7f1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:00.523488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1020/2044] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:34:00.525188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca216e8-372d-4cad-840d-1375085076e6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:00.535906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca216e8-372d-4cad-840d-1375085076e6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:00.541369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:00.541980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.542401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.542515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.543199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:00.543500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:00.545148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:00.563188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed300c74-3b70-4f03-853c-fa61b4a7f1df None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:00.563404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1025/2045] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:00.573105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed300c74-3b70-4f03-853c-fa61b4a7f1df None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:00.577995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:00.578612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.578888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.579198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.579868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:00.583130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:00.617058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1021/2046] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:00 2022] POST /identity/v3/projects => generated 379 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:34:00.621654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:00.631631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebd4c8-d996-4822-bf8d-a5b216f5370a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:00.690810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:00.691364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.691644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.691911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.692544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:00.694808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:00.720239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1026/2047] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:00 2022] POST /identity/v3/users => generated 290 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:00.724441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:00.736198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c541947-d9b3-4ea7-8956-eae78e20f155 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:00.742044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:00.742642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.743203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.743473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.744111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:00.745690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:00.756549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1022/2048] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:00.761372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:00.773377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bffedae-23b3-4fc4-8605-abf38d58976e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:00.778162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9370b8-79f2-42ca-bce7-1cb464050cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:00.778764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9370b8-79f2-42ca-bce7-1cb464050cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.779072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9370b8-79f2-42ca-bce7-1cb464050cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.779345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9370b8-79f2-42ca-bce7-1cb464050cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.779983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9370b8-79f2-42ca-bce7-1cb464050cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:00.781589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9370b8-79f2-42ca-bce7-1cb464050cb9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:00.792524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1027/2049] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:00.796825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:00.806721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3495b5-2e16-4a2a-bd05-dd47a58fda9d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:00.811574 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:00.812219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.812526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.812803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] PATH_INFO: `/v3/projects/d951f19622f24014b0db552eafd4346d/users/1bd18015d8334028ada90e42558d7942/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.827947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d951f19622f24014b0db552eafd4346d, user_id=1bd18015d8334028ada90e42558d7942, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:00.830588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:00.851718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1023/2050] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:00 2022] PUT /identity/v3/projects/d951f19622f24014b0db552eafd4346d/users/1bd18015d8334028ada90e42558d7942/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:00.853556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24588578-1c37-418d-b5c7-06b0431addb1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:00.853846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24588578-1c37-418d-b5c7-06b0431addb1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:00.854120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24588578-1c37-418d-b5c7-06b0431addb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:00.884028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-24588578-1c37-418d-b5c7-06b0431addb1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:00.885398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-24588578-1c37-418d-b5c7-06b0431addb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:00.979433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1028/2051] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:01.001008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:01.003485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:01.038505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:01.091744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae00529e-5066-4bf4-aebf-0d471ba4e865 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:01.096247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:01.097013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:01.097286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:01.097616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:01.109828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:01.112379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:01.130823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1024/2052] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:01.959005 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:01.959549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:01.959549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:02.005029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] MFA Rules not processed for user `1bd18015d8334028ada90e42558d7942`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:02.006476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:02.044692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:02.110211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1029/2053] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:01 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:02.125177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:02.127288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:02.200181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8cae16-e4c7-4741-86fc-bc0dde97756c service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:02.204923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:02.206175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:02.206533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:02.206875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:02.224681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:02.227230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:02.237852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1025/2054] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:02 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:02.509668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:02.511486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:02.534673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a026a-8c6c-4ce2-97cb-764ae877d7de None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:02.541392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:02.542190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:02.542755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:02.543185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] PATH_INFO: `/v3/users/4f29428b32454dbdb9c644759389abec` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:02.544566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f29428b32454dbdb9c644759389abec)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:02.546722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:02.618949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f29428b32454dbdb9c644759389abec'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:02.623537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f29428b32454dbdb9c644759389abec'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:02.639179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f29428b32454dbdb9c644759389abec'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:02.639179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f29428b32454dbdb9c644759389abec'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:02.655267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23396213-52b3-49ca-a60e-3924303048fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f29428b32454dbdb9c644759389abec'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:02.678936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1030/2055] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:02 2022] DELETE /identity/v3/users/4f29428b32454dbdb9c644759389abec => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:02.931079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:02.950851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b17c52-1219-4533-965a-000e4b79cd34 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.011426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.012052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.012332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.012606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] PATH_INFO: `/v3/projects/93b9baf342f94f34ae8f649db7198137` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.013812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] RBAC: Authorizing `identity:delete_project(project_id=93b9baf342f94f34ae8f649db7198137)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.015795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.071489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93b9baf342f94f34ae8f649db7198137', 'request_id': 'req-b07b96f6-8a25-4204-9987-413e3ece363f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:03.077761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93b9baf342f94f34ae8f649db7198137', 'request_id': 'req-b07b96f6-8a25-4204-9987-413e3ece363f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:03.083356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93b9baf342f94f34ae8f649db7198137', 'request_id': 'req-b07b96f6-8a25-4204-9987-413e3ece363f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:03.087115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:03.098726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] The token cache is being invalidate because project 93b9baf342f94f34ae8f649db7198137 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:03.099041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:03.099664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b07b96f6-8a25-4204-9987-413e3ece363f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:03.101680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1026/2056] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:02 2022] DELETE /identity/v3/projects/93b9baf342f94f34ae8f649db7198137 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:03.106471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d77e9-6824-4443-bf96-6689ff3db4f3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.106927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d77e9-6824-4443-bf96-6689ff3db4f3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.107058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d77e9-6824-4443-bf96-6689ff3db4f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.128856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9a9d77e9-6824-4443-bf96-6689ff3db4f3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:03.130059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9a9d77e9-6824-4443-bf96-6689ff3db4f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:03.187706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1031/2057] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:03.195470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5136be3-9a86-4f56-b53c-78e54029d5d6 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.195959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5136be3-9a86-4f56-b53c-78e54029d5d6 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.196064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5136be3-9a86-4f56-b53c-78e54029d5d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.215421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b5136be3-9a86-4f56-b53c-78e54029d5d6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:03.216608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b5136be3-9a86-4f56-b53c-78e54029d5d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:03.224772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1027/2058] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:03.229367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d77e9-6824-4443-bf96-6689ff3db4f3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:03.239320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d77e9-6824-4443-bf96-6689ff3db4f3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.245113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.245706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.245980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.246246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.246873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:03.247195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.248777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.260312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1032/2059] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:03.264481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5136be3-9a86-4f56-b53c-78e54029d5d6 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:03.273547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5136be3-9a86-4f56-b53c-78e54029d5d6 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.278092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.278672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.278945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.279240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.279870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.281732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.309685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1028/2060] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:03 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:03.313948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:03.324044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db4ebe2-f97f-4ce0-9cf2-97c50cd72ea3 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.336026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:03.382281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.382878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.383205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.383475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.384116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.386010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.407634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1033/2061] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:03 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:03.412100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:03.423039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a38b7b3-f722-46bc-a001-9ed7cca74c25 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.427679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.428258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.428686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.428929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.429559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.431155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.440819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1029/2062] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:03.444720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:03.454455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff31f7e-5f6a-483f-a208-dfc9ecdbc772 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.459132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8889b182-a355-4c25-aae4-e46278957789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.460206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8889b182-a355-4c25-aae4-e46278957789 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.460482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8889b182-a355-4c25-aae4-e46278957789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.460759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8889b182-a355-4c25-aae4-e46278957789 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.461411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8889b182-a355-4c25-aae4-e46278957789 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.463009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8889b182-a355-4c25-aae4-e46278957789 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.473418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1034/2063] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:03.478157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:03.487789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c355b1c0-c288-4b8f-8d07-161ecc85dc86 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.492468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.493344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.493615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.493887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] PATH_INFO: `/v3/projects/e0af60ab182841269cee39048e22984d/users/b4d39c850b024bde8e69665a85f09d8a/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.508048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0af60ab182841269cee39048e22984d, user_id=b4d39c850b024bde8e69665a85f09d8a, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.510684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.523935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1030/2064] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:03 2022] PUT /identity/v3/projects/e0af60ab182841269cee39048e22984d/users/b4d39c850b024bde8e69665a85f09d8a/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:03.533934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189fbee-eedf-4af8-be1a-5df53e7a0e4d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.534233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189fbee-eedf-4af8-be1a-5df53e7a0e4d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.534508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2189fbee-eedf-4af8-be1a-5df53e7a0e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.562010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-2189fbee-eedf-4af8-be1a-5df53e7a0e4d None None] MFA Rules not processed for user `b4d39c850b024bde8e69665a85f09d8a`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:03.563236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2189fbee-eedf-4af8-be1a-5df53e7a0e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:03.639142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1035/2065] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:03.647050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:03.649419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:03.728875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85519dc0-1729-46df-bd9a-8991706463bb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:03.734772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:03.735561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.735841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.736127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.745843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:03.748005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:03.758495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1031/2066] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:03 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:03.790840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61a007-89ac-4a93-b1e1-7c97fd949623 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:03.791158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61a007-89ac-4a93-b1e1-7c97fd949623 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:03.791432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61a007-89ac-4a93-b1e1-7c97fd949623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:03.810528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3d61a007-89ac-4a93-b1e1-7c97fd949623 None None] MFA Rules not processed for user `1bd18015d8334028ada90e42558d7942`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:03.811742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3d61a007-89ac-4a93-b1e1-7c97fd949623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:03.866896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1036/2067] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:03 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:13.673545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:13.735445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dbdc98-ed7f-4aa4-9fbd-364ff89ecb55 service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:13.739925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:13.740701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:13.741501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:13.741775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] PATH_INFO: `/v3/users/b4d39c850b024bde8e69665a85f09d8a` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:13.743210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4d39c850b024bde8e69665a85f09d8a)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:13.745123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:13.772009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:13.798370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4d39c850b024bde8e69665a85f09d8a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:13.799149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4d39c850b024bde8e69665a85f09d8a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:13.807335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4d39c850b024bde8e69665a85f09d8a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:13.818788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4d39c850b024bde8e69665a85f09d8a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:13.819366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4d39c850b024bde8e69665a85f09d8a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:13.832771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1032/2068] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:13 2022] DELETE /identity/v3/users/b4d39c850b024bde8e69665a85f09d8a => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:13.837609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76465524-7d90-483c-96aa-5ee9c9eb55ce None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:13.838077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76465524-7d90-483c-96aa-5ee9c9eb55ce None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:13.838287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76465524-7d90-483c-96aa-5ee9c9eb55ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:13.858394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-76465524-7d90-483c-96aa-5ee9c9eb55ce None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:13.859757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-76465524-7d90-483c-96aa-5ee9c9eb55ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:13.887554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:13.922100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1037/2069] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:13.931968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:13.933484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:13.973363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a13d6-a173-442b-b517-5519a0281213 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:13.973677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a13d6-a173-442b-b517-5519a0281213 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:13.973987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a13d6-a173-442b-b517-5519a0281213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:13.996325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3f5a13d6-a173-442b-b517-5519a0281213 None None] MFA Rules not processed for user `db187750f7514609923133b9a5f17ae7`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:13.997439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3f5a13d6-a173-442b-b517-5519a0281213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:13.997655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2a0625-eefb-490d-a09b-dd553ff96899 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.001256 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.001853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.002125 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.002395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.012428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.013854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.024654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1033/2070] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:14.057516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1038/2071] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:34:14.304967 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:14.306469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:14.320688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8745158-c1d8-4b8b-8307-25d63e2edb5f service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.327944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.328583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.328856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.329121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] PATH_INFO: `/v3/projects/e0af60ab182841269cee39048e22984d` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.330232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0af60ab182841269cee39048e22984d)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.333685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.384811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0af60ab182841269cee39048e22984d', 'request_id': 'req-3e593bb5-7c74-4882-a649-e66b9c987402'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:14.390869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0af60ab182841269cee39048e22984d', 'request_id': 'req-3e593bb5-7c74-4882-a649-e66b9c987402'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:14.396662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0af60ab182841269cee39048e22984d', 'request_id': 'req-3e593bb5-7c74-4882-a649-e66b9c987402'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:14.410836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] The token cache is being invalidate because project e0af60ab182841269cee39048e22984d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:14.411164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:14.411837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3e593bb5-7c74-4882-a649-e66b9c987402 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:14.414535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1034/2072] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:14 2022] DELETE /identity/v3/projects/e0af60ab182841269cee39048e22984d => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:14.419365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644578bf-3c17-4e73-8ebb-1ca82a219e87 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.419651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644578bf-3c17-4e73-8ebb-1ca82a219e87 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.419928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644578bf-3c17-4e73-8ebb-1ca82a219e87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.441759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-644578bf-3c17-4e73-8ebb-1ca82a219e87 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:14.443023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-644578bf-3c17-4e73-8ebb-1ca82a219e87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:14.504757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1039/2073] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:14.512655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e83547-d494-4662-bdf7-26db57ba257f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.512938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e83547-d494-4662-bdf7-26db57ba257f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.513212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e83547-d494-4662-bdf7-26db57ba257f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.534075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-79e83547-d494-4662-bdf7-26db57ba257f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:14.535432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-79e83547-d494-4662-bdf7-26db57ba257f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:14.543500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1035/2074] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:14.547786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644578bf-3c17-4e73-8ebb-1ca82a219e87 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:14.559705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644578bf-3c17-4e73-8ebb-1ca82a219e87 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.564171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.564763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.565147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.565420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.566056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:14.566353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.567964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.581151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1040/2075] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:14.586692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e83547-d494-4662-bdf7-26db57ba257f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:14.596616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e83547-d494-4662-bdf7-26db57ba257f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.602408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.603039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.603322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.603596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.604233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.606184 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.623962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:14.643888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1036/2076] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:14 2022] POST /identity/v3/projects => generated 403 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:14.644989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:14.656383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d55860c-0e74-4c36-bad3-649997d9c6bd admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.737347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.737959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.738232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.738501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.739164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.742057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.755751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:14.773112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1041/2077] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:14 2022] POST /identity/v3/users => generated 302 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:14.776843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:14.789169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310aca4d-36f5-4d1e-a259-10dfbc060bcc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.796543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.797424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.797947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.798421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.800141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.802565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.817876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1037/2078] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:14 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:14.822009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:14.836904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92d561d-218f-427c-8711-9e27adc341fd admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.847756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc06b2c-5d76-4018-b9ee-983818412de2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.848463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc06b2c-5d76-4018-b9ee-983818412de2 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.848615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc06b2c-5d76-4018-b9ee-983818412de2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.848890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc06b2c-5d76-4018-b9ee-983818412de2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.849524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc06b2c-5d76-4018-b9ee-983818412de2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.852090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc06b2c-5d76-4018-b9ee-983818412de2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.863549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1042/2079] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:14 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:34:14.868331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:14.880344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef01b737-55ec-4cdc-a9f5-5c513aa4b7cf admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:14.885585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:14.886266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.886539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.886809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] PATH_INFO: `/v3/projects/4857661ebb304a82a36555d31da019c1/users/bb3f33a7941444659559aed797f41222/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.904422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4857661ebb304a82a36555d31da019c1, user_id=bb3f33a7941444659559aed797f41222, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:14.907180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:14.925329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1038/2080] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:14 2022] PUT /identity/v3/projects/4857661ebb304a82a36555d31da019c1/users/bb3f33a7941444659559aed797f41222/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:34:14.930483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7cc19-167d-45df-822b-2db9976aaa08 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:14.931118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7cc19-167d-45df-822b-2db9976aaa08 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:14.931422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7cc19-167d-45df-822b-2db9976aaa08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:14.953287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-cbf7cc19-167d-45df-822b-2db9976aaa08 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:14.955213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cbf7cc19-167d-45df-822b-2db9976aaa08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:15.026316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1043/2081] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:15.038431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:15.041081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:15.112663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c37e6a0-6a32-44d5-be03-ef644a0b5d44 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:15.117158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:15.117758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:15.118307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:15.118581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:15.127575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:15.128991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:15.139102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1039/2082] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:15.766345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3f364c-63e3-44fc-aa62-2bccff1d232f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:15.766791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3f364c-63e3-44fc-aa62-2bccff1d232f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:15.766907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3f364c-63e3-44fc-aa62-2bccff1d232f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:15.798618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-0e3f364c-63e3-44fc-aa62-2bccff1d232f None None] MFA Rules not processed for user `bb3f33a7941444659559aed797f41222`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:15.800691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0e3f364c-63e3-44fc-aa62-2bccff1d232f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:15.881341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1044/2083] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:15 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:15.885756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:15.900341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2fecdd-64b0-4926-9aed-a79525ea2092 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:15.910549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:15.911546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:15.913211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:15.913800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:15.914606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:15.917311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:15.952857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1040/2084] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:15 2022] POST /identity/v3/projects => generated 405 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:15.957075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3f364c-63e3-44fc-aa62-2bccff1d232f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:15.973701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3f364c-63e3-44fc-aa62-2bccff1d232f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:16.044991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:16.045619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:16.045903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:16.046172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:16.046805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:16.048822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:16.069890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1045/2085] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:15 2022] POST /identity/v3/users => generated 309 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:34:16.074006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:16.084288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0346b21c-0c59-4612-bc2e-4f8e5b27b3f7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:16.089281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:16.089862 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:16.090138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:16.090405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:16.091072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:16.092647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:16.105760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1041/2086] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:16.109923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:16.115008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:16.122328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fcb95a-732b-4403-909d-142e385d8dab admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:16.127839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:16.128510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:16.128786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:16.129054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] PATH_INFO: `/v3/projects/ab5dfa6495f146518eab5ed81717fba0/users/330f7df9436f47d193e4b25e7d73cfa8/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:16.144578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab5dfa6495f146518eab5ed81717fba0, user_id=330f7df9436f47d193e4b25e7d73cfa8, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:16.147976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:16.161172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1046/2087] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:16 2022] PUT /identity/v3/projects/ab5dfa6495f146518eab5ed81717fba0/users/330f7df9436f47d193e4b25e7d73cfa8/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:34:16.166387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:16.176480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43deb1-4050-4053-b0fd-e03223bf56c8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:16.246894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bec4b5a-c4e6-43ee-90ff-558213286004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:16.247641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bec4b5a-c4e6-43ee-90ff-558213286004 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:16.247921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bec4b5a-c4e6-43ee-90ff-558213286004 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:16.248194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bec4b5a-c4e6-43ee-90ff-558213286004 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:16.248841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bec4b5a-c4e6-43ee-90ff-558213286004 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:16.250436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bec4b5a-c4e6-43ee-90ff-558213286004 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:16.262139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1042/2088] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:16 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:16.266272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:16.290899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb793a63-7464-489e-a4d1-192a2b7ceae0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:16.296968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:16.297635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:16.297914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:16.298189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] PATH_INFO: `/v3/projects/ab5dfa6495f146518eab5ed81717fba0/users/330f7df9436f47d193e4b25e7d73cfa8/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:16.300316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab5dfa6495f146518eab5ed81717fba0, user_id=330f7df9436f47d193e4b25e7d73cfa8, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:16.303155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:16.314528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1047/2089] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:16 2022] PUT /identity/v3/projects/ab5dfa6495f146518eab5ed81717fba0/users/330f7df9436f47d193e4b25e7d73cfa8/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:17.018281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba60f1-96cb-4949-9277-5447bdf9cb60 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:17.018784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba60f1-96cb-4949-9277-5447bdf9cb60 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:17.018951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba60f1-96cb-4949-9277-5447bdf9cb60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:17.049047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2fba60f1-96cb-4949-9277-5447bdf9cb60 None None] MFA Rules not processed for user `330f7df9436f47d193e4b25e7d73cfa8`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:17.050413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2fba60f1-96cb-4949-9277-5447bdf9cb60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:17.066255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:17.139351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1043/2090] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:17 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:18.813159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:18.874739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf6e3f-6b6f-433c-ae2e-61d120aec640 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:18.879214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:18.879819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:18.880092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:18.880361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] PATH_INFO: `/v3/users/bb3f33a7941444659559aed797f41222` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:18.881598 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb3f33a7941444659559aed797f41222)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:18.883445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:18.934255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb3f33a7941444659559aed797f41222'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:18.934899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb3f33a7941444659559aed797f41222'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:18.942281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb3f33a7941444659559aed797f41222'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:18.942857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb3f33a7941444659559aed797f41222'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:18.955599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb3f33a7941444659559aed797f41222'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:18.970841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1048/2091] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:18 2022] DELETE /identity/v3/users/bb3f33a7941444659559aed797f41222 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:19.153180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba60f1-96cb-4949-9277-5447bdf9cb60 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:19.167715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba60f1-96cb-4949-9277-5447bdf9cb60 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:19.230612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:19.231277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:19.231560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:19.231837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] PATH_INFO: `/v3/projects/4857661ebb304a82a36555d31da019c1` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:19.232984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4857661ebb304a82a36555d31da019c1)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:19.234891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:19.283635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4857661ebb304a82a36555d31da019c1', 'request_id': 'req-e7aab8cc-79b1-47d4-abea-cf98c20995f9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.290007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4857661ebb304a82a36555d31da019c1', 'request_id': 'req-e7aab8cc-79b1-47d4-abea-cf98c20995f9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.295562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4857661ebb304a82a36555d31da019c1', 'request_id': 'req-e7aab8cc-79b1-47d4-abea-cf98c20995f9'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.308127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] The token cache is being invalidate because project 4857661ebb304a82a36555d31da019c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:19.308416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.309024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.310516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1044/2092] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:19 2022] DELETE /identity/v3/projects/4857661ebb304a82a36555d31da019c1 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:19.314599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:19.316007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:19.329818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c1223c-ebe4-43ce-8b36-5c4d8c875191 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:19.370813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:19.387161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:19.387779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:19.388052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:19.388323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] PATH_INFO: `/v3/users/330f7df9436f47d193e4b25e7d73cfa8` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:19.389408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=330f7df9436f47d193e4b25e7d73cfa8)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:19.391354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:19.438351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '330f7df9436f47d193e4b25e7d73cfa8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.439011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '330f7df9436f47d193e4b25e7d73cfa8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.463876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '330f7df9436f47d193e4b25e7d73cfa8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.464481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '330f7df9436f47d193e4b25e7d73cfa8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.480630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2d719675-f193-48c7-9162-9d34dcc4bbbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '330f7df9436f47d193e4b25e7d73cfa8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.495453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1049/2093] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:19 2022] DELETE /identity/v3/users/330f7df9436f47d193e4b25e7d73cfa8 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:19.668583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:19.670245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:19.689147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7aab8cc-79b1-47d4-abea-cf98c20995f9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:19.734480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:19.756560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:19.757190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:19.757464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:19.757731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] PATH_INFO: `/v3/projects/ab5dfa6495f146518eab5ed81717fba0` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:19.758920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab5dfa6495f146518eab5ed81717fba0)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:19.760925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:19.819974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab5dfa6495f146518eab5ed81717fba0', 'request_id': 'req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.827557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab5dfa6495f146518eab5ed81717fba0', 'request_id': 'req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.833767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab5dfa6495f146518eab5ed81717fba0', 'request_id': 'req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.853607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] The token cache is being invalidate because project ab5dfa6495f146518eab5ed81717fba0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:19.853900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.854547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0cc10930-af4f-4bf1-92b5-21cd4eafb69b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:19.859563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1045/2094] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:19 2022] DELETE /identity/v3/projects/ab5dfa6495f146518eab5ed81717fba0 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:19.861257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb5fd0c-7cf2-4740-ab4d-d279abcee278 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:19.861734 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb5fd0c-7cf2-4740-ab4d-d279abcee278 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:19.861820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb5fd0c-7cf2-4740-ab4d-d279abcee278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:19.883706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-cfb5fd0c-7cf2-4740-ab4d-d279abcee278 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:19.884908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cfb5fd0c-7cf2-4740-ab4d-d279abcee278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:19.960189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1050/2095] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:19.962060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18df8ba-be4e-4fc6-b7a3-ba33bab32d6e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:19.962353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18df8ba-be4e-4fc6-b7a3-ba33bab32d6e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:19.962806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18df8ba-be4e-4fc6-b7a3-ba33bab32d6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:19.986784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c18df8ba-be4e-4fc6-b7a3-ba33bab32d6e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:19.987752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c18df8ba-be4e-4fc6-b7a3-ba33bab32d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:19.996742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1046/2096] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:20.001162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb5fd0c-7cf2-4740-ab4d-d279abcee278 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.011670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb5fd0c-7cf2-4740-ab4d-d279abcee278 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.016657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.017368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.017751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.018254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.019040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:20.019426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.021092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.034018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1051/2097] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:20.039333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18df8ba-be4e-4fc6-b7a3-ba33bab32d6e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.051686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18df8ba-be4e-4fc6-b7a3-ba33bab32d6e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.056999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.057626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.057878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.058131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.058919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.060997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.096786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1047/2098] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:20 2022] POST /identity/v3/projects => generated 379 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:20.098359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.113816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220a06e2-ec7a-4527-bea8-8000c19c3825 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.168130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:20.192609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.193224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.193586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.193908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.194726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.196845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.235329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1052/2099] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:20 2022] POST /identity/v3/users => generated 290 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:20.239478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.255737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7129d46f-1b1d-4503-9bbe-dd8c995ca786 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.261636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.262333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.262899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.263062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.264582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.266246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.282938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1048/2100] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:20.286636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.301865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116fab93-fe44-403d-8205-a0d0270df5ff admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.308243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40032ed-5b5f-4a94-bf76-9673107abaca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.309084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40032ed-5b5f-4a94-bf76-9673107abaca admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.309457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40032ed-5b5f-4a94-bf76-9673107abaca admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.309819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40032ed-5b5f-4a94-bf76-9673107abaca admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.310887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40032ed-5b5f-4a94-bf76-9673107abaca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.313246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40032ed-5b5f-4a94-bf76-9673107abaca admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.333334 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1053/2101] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:34:20.339543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.357511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8247fb7-9df3-40ba-8ddb-d99c51bf101d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.363342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.364260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.364636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.365050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] PATH_INFO: `/v3/projects/38bf58eec49840179ddb2d635da67234/users/2ff5a9dc99f34024aba66c16da6ae1c6/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.388491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38bf58eec49840179ddb2d635da67234, user_id=2ff5a9dc99f34024aba66c16da6ae1c6, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.391281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.412039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1049/2102] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:20 2022] PUT /identity/v3/projects/38bf58eec49840179ddb2d635da67234/users/2ff5a9dc99f34024aba66c16da6ae1c6/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:20.422625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e2735-1bf7-48a2-865b-eba11917e571 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.423048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e2735-1bf7-48a2-865b-eba11917e571 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.423339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e2735-1bf7-48a2-865b-eba11917e571 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.452498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5b7e2735-1bf7-48a2-865b-eba11917e571 None None] MFA Rules not processed for user `2ff5a9dc99f34024aba66c16da6ae1c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:20.453695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5b7e2735-1bf7-48a2-865b-eba11917e571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:20.516499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1054/2103] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:20.520838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.580650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829864e8-a9bb-49d3-8a18-afda34d7a280 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.585883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.586465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.586739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.587035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.587650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.589520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.600337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:20.617507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1050/2104] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:20 2022] POST /identity/v3/projects => generated 381 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:20.621870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e2735-1bf7-48a2-865b-eba11917e571 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.633191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e2735-1bf7-48a2-865b-eba11917e571 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.688909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.689499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.689771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.690037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.690661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.692633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.717016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1055/2105] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:20 2022] POST /identity/v3/users => generated 297 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:20.718664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.730124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3c16ad-55d3-4042-a3e6-89d0792e891e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.734695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.735311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.735586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.735856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.736484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.738070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.748425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1051/2106] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:20.752850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.762620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afa1477-ecb9-4f04-8ca2-a72b3591d7fb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.767199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.767794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.768068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.768338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] PATH_INFO: `/v3/projects/0a6c7f8c95144189bfc80b4d98908e1f/users/dbc7bf8d101e4c10b863dfde0081c8f3/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.783412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a6c7f8c95144189bfc80b4d98908e1f, user_id=dbc7bf8d101e4c10b863dfde0081c8f3, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.786143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.799552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1056/2107] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:20 2022] PUT /identity/v3/projects/0a6c7f8c95144189bfc80b4d98908e1f/users/dbc7bf8d101e4c10b863dfde0081c8f3/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:20.804568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.815144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd628ba2-789b-4478-9277-e60115bbe06d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.873089 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c5074d-c553-4de9-9f1e-531a37e24d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.873697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5074d-c553-4de9-9f1e-531a37e24d50 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.873988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5074d-c553-4de9-9f1e-531a37e24d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.874246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c5074d-c553-4de9-9f1e-531a37e24d50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.874883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c5074d-c553-4de9-9f1e-531a37e24d50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.876531 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c5074d-c553-4de9-9f1e-531a37e24d50 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.887899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1052/2108] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:20 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:20.891931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:20.902332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53f939b-5b3f-4e3a-b1ce-56a3be9d2c7a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:20.906923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:20.907592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.907866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.908140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] PATH_INFO: `/v3/projects/0a6c7f8c95144189bfc80b4d98908e1f/users/dbc7bf8d101e4c10b863dfde0081c8f3/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.910015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a6c7f8c95144189bfc80b4d98908e1f, user_id=dbc7bf8d101e4c10b863dfde0081c8f3, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:20.912801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:20.922665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1057/2109] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:20 2022] PUT /identity/v3/projects/0a6c7f8c95144189bfc80b4d98908e1f/users/dbc7bf8d101e4c10b863dfde0081c8f3/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:20.934342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72ac71-5465-41d7-8f13-9e09f0329a1e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:20.934625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72ac71-5465-41d7-8f13-9e09f0329a1e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:20.934899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f72ac71-5465-41d7-8f13-9e09f0329a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:20.965285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8f72ac71-5465-41d7-8f13-9e09f0329a1e None None] MFA Rules not processed for user `dbc7bf8d101e4c10b863dfde0081c8f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:20.966741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8f72ac71-5465-41d7-8f13-9e09f0329a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:21.049117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1053/2110] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:20 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:21.062736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:21.064557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:21.082480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:21.142212 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2ee271-308f-49b7-a550-68a14f7e17ca admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:21.148348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:21.149078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:21.149359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:21.149681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:21.161741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:21.163543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:21.175660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1058/2111] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:21 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:21.587914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72ac71-5465-41d7-8f13-9e09f0329a1e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:21.602989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f72ac71-5465-41d7-8f13-9e09f0329a1e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:21.608636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbc7bf8d101e4c10b863dfde0081c8f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0a6c7f8c95144189bfc80b4d98908e1f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:21.610062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:21.610534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:21.610990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:21.611814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:21.612314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:21.614892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:21.633906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1054/2112] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:21.638280 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:21.649633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cbb25-21b8-4e1b-9bae-0594ec41d97c service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:21.655780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbc7bf8d101e4c10b863dfde0081c8f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0a6c7f8c95144189bfc80b4d98908e1f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:21.656394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:21.656677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:21.656986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:21.658258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:21.660581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:21.694508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1059/2113] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:21 2022] POST /identity/v3/projects => generated 353 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:34:21.696068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:21.713931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfaa580-d12c-4690-a750-a1303a4e6c22 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:21.787368 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:21.804741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbc7bf8d101e4c10b863dfde0081c8f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0a6c7f8c95144189bfc80b4d98908e1f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:21.805346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:21.805675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:21.805907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:21.806556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:21.806979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:21.808663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:21.826252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1055/2114] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:21.869640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:21.881680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3970bee5-7c7c-45ff-aee5-65179fbdc681 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:21.886318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dbc7bf8d101e4c10b863dfde0081c8f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0a6c7f8c95144189bfc80b4d98908e1f', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:21.886985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:21.887273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:21.887544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] PATH_INFO: `/v3/projects/4bf2814566fa40c8b7128dbe78ce1371` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:21.888654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorizing `identity:delete_project(project_id=4bf2814566fa40c8b7128dbe78ce1371)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:21.890562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:21.938568 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4bf2814566fa40c8b7128dbe78ce1371', 'request_id': 'req-5f2eaac5-7638-4d84-8f80-d62248cc49c7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:21.944940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4bf2814566fa40c8b7128dbe78ce1371', 'request_id': 'req-5f2eaac5-7638-4d84-8f80-d62248cc49c7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:21.951980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4bf2814566fa40c8b7128dbe78ce1371', 'request_id': 'req-5f2eaac5-7638-4d84-8f80-d62248cc49c7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:21.965364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] The token cache is being invalidate because project 4bf2814566fa40c8b7128dbe78ce1371 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:21.965660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:21.966366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5f2eaac5-7638-4d84-8f80-d62248cc49c7 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:21.968475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1060/2115] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:21 2022] DELETE /identity/v3/projects/4bf2814566fa40c8b7128dbe78ce1371 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:22.175105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:22.176772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:22.254015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ba16a-4f9d-4560-a782-bd1299adfab1 tempest-VolumeQuotaClassesTest-1931178893 tempest-VolumeQuotaClassesTest-1931178893-project-admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:22.316653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:22.317180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:22.317453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:22.317721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] PATH_INFO: `/v3/users/2ff5a9dc99f34024aba66c16da6ae1c6` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:22.319430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ff5a9dc99f34024aba66c16da6ae1c6)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:22.321578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:22.376738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ff5a9dc99f34024aba66c16da6ae1c6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:22.377590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ff5a9dc99f34024aba66c16da6ae1c6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:22.385744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ff5a9dc99f34024aba66c16da6ae1c6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:22.397585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ff5a9dc99f34024aba66c16da6ae1c6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:22.398237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ff5a9dc99f34024aba66c16da6ae1c6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:22.410817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1056/2116] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:22 2022] DELETE /identity/v3/users/2ff5a9dc99f34024aba66c16da6ae1c6 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:22.415387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:22.415666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:22.415934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:22.435943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:22.437533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:22.497918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1061/2117] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:22.507865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:22.509290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:22.523441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:22.573902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad827731-2d56-49ac-8904-10f61c81be0e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:22.577659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:22.578255 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:22.578533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:22.578800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:22.588106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:22.589524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:22.599210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1057/2118] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:23.053802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:23.055542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:23.068666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9191f72b-348c-4d22-8c81-38b484c0bcf4 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:23.080316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:23.081025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:23.081375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:23.081717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] PATH_INFO: `/v3/projects/38bf58eec49840179ddb2d635da67234` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:23.082926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] RBAC: Authorizing `identity:delete_project(project_id=38bf58eec49840179ddb2d635da67234)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:23.085654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:23.097804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:23.144171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38bf58eec49840179ddb2d635da67234', 'request_id': 'req-708b579b-fefe-4b11-beac-a9073db53483'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.150289 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38bf58eec49840179ddb2d635da67234', 'request_id': 'req-708b579b-fefe-4b11-beac-a9073db53483'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.156885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38bf58eec49840179ddb2d635da67234', 'request_id': 'req-708b579b-fefe-4b11-beac-a9073db53483'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.170274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] The token cache is being invalidate because project 38bf58eec49840179ddb2d635da67234 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:23.170557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.171191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.172677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1062/2119] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:23 2022] DELETE /identity/v3/projects/38bf58eec49840179ddb2d635da67234 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:23.176610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:23.186084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a864d2-009f-4041-b14b-ef2e99f2887d service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:23.247013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:23.247631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:23.247905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:23.248172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] PATH_INFO: `/v3/users/dbc7bf8d101e4c10b863dfde0081c8f3` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:23.249266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbc7bf8d101e4c10b863dfde0081c8f3)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:23.251222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:23.300179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbc7bf8d101e4c10b863dfde0081c8f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.300815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbc7bf8d101e4c10b863dfde0081c8f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.308033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbc7bf8d101e4c10b863dfde0081c8f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.320542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbc7bf8d101e4c10b863dfde0081c8f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.321178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7e61c1f1-48ce-4cef-9576-0f623fb4aa4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbc7bf8d101e4c10b863dfde0081c8f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.333716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1058/2120] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:23 2022] DELETE /identity/v3/users/dbc7bf8d101e4c10b863dfde0081c8f3 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:23.611588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:23.613002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:23.632969 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708b579b-fefe-4b11-beac-a9073db53483 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:23.690281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:23.690905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:23.691210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:23.691484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] PATH_INFO: `/v3/projects/0a6c7f8c95144189bfc80b4d98908e1f` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:23.692591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a6c7f8c95144189bfc80b4d98908e1f)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:23.695221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:23.742847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a6c7f8c95144189bfc80b4d98908e1f', 'request_id': 'req-1da16e98-b713-43f2-936e-bedf257a2aba'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.748579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a6c7f8c95144189bfc80b4d98908e1f', 'request_id': 'req-1da16e98-b713-43f2-936e-bedf257a2aba'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.757323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a6c7f8c95144189bfc80b4d98908e1f', 'request_id': 'req-1da16e98-b713-43f2-936e-bedf257a2aba'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.773922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] The token cache is being invalidate because project 0a6c7f8c95144189bfc80b4d98908e1f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:23.774200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.774814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1da16e98-b713-43f2-936e-bedf257a2aba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:23.776343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1063/2121] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:23 2022] DELETE /identity/v3/projects/0a6c7f8c95144189bfc80b4d98908e1f => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:23.781285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216724c-2fd7-4170-b5fc-ec428e61a3c2 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:23.781774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216724c-2fd7-4170-b5fc-ec428e61a3c2 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:23.781862 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216724c-2fd7-4170-b5fc-ec428e61a3c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:23.801519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6216724c-2fd7-4170-b5fc-ec428e61a3c2 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:23.803337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6216724c-2fd7-4170-b5fc-ec428e61a3c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:23.837790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:23.883211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1059/2122] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:23.891873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e92fdc-63f2-4c79-9ee6-b72dd343fd7a None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:23.891873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e92fdc-63f2-4c79-9ee6-b72dd343fd7a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:23.891873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e92fdc-63f2-4c79-9ee6-b72dd343fd7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:23.911841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-39e92fdc-63f2-4c79-9ee6-b72dd343fd7a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:23.913379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-39e92fdc-63f2-4c79-9ee6-b72dd343fd7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:23.922536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1064/2123] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:23.926633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216724c-2fd7-4170-b5fc-ec428e61a3c2 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:23.937606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216724c-2fd7-4170-b5fc-ec428e61a3c2 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:23.942208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:23.942801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:23.943100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:23.944166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:23.944808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:23.945107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:23.946827 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:23.958643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1060/2124] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:34:23.963271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e92fdc-63f2-4c79-9ee6-b72dd343fd7a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:23.975728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e92fdc-63f2-4c79-9ee6-b72dd343fd7a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:23.985305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:23.985920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:23.986193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:23.986459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:23.987134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:23.989034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.010942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:24.018076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1065/2125] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:23 2022] POST /identity/v3/projects => generated 381 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:24.022439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.032793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dd7075-7711-4708-9b5e-bd16c4a9c752 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.090246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.090847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.091153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.091423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.092042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.094092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.118299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1061/2126] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:24 2022] POST /identity/v3/users => generated 291 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:24.122524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.132908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b2751-e5bc-4f86-b994-b28016f89c2a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.137542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.138132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.138404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.138669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.139354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.140921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.156422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1066/2127] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:34:24.158055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.169998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a549da80-a428-4e0a-9262-4b57a5528e0f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.175013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8baf8f68-6336-418d-938d-1038a45271f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.175610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baf8f68-6336-418d-938d-1038a45271f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.175891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baf8f68-6336-418d-938d-1038a45271f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.176161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8baf8f68-6336-418d-938d-1038a45271f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.176800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8baf8f68-6336-418d-938d-1038a45271f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.179390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8baf8f68-6336-418d-938d-1038a45271f0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.190818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1062/2128] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:24.196046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.206293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982af50e-8fb8-4d36-9b93-49653a8280b9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.212184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.213449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.213729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.214077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] PATH_INFO: `/v3/projects/ea39567800cb4c71aabb51a765f8bc18/users/0c214914259a416d9a163d6912434994/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.236152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea39567800cb4c71aabb51a765f8bc18, user_id=0c214914259a416d9a163d6912434994, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.238821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.253611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1067/2129] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:24 2022] PUT /identity/v3/projects/ea39567800cb4c71aabb51a765f8bc18/users/0c214914259a416d9a163d6912434994/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:24.265140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e091c02-0310-45a9-a605-954c592a96a4 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.265436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e091c02-0310-45a9-a605-954c592a96a4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.265706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e091c02-0310-45a9-a605-954c592a96a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.297699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8e091c02-0310-45a9-a605-954c592a96a4 None None] MFA Rules not processed for user `0c214914259a416d9a163d6912434994`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:24.298894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8e091c02-0310-45a9-a605-954c592a96a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:24.360926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1063/2130] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:24 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:24.366273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.428254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed616c94-8a6a-427e-9ea2-4733742066a7 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.432736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.433351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.433653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.433945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.434563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.436497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.464108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1068/2131] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:24 2022] POST /identity/v3/projects => generated 379 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:24.468376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e091c02-0310-45a9-a605-954c592a96a4 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.480331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e091c02-0310-45a9-a605-954c592a96a4 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.504025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:24.538585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.539286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.539595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.539887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.540538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.542623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.565989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1064/2132] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:24 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:24.570262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.580716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0038f1-8979-4f72-9d16-d21ffd0263a5 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.586394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.587071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.587353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.587655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.588296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.589962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.602297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1069/2133] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:24.606281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.616214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4012e2d0-48b9-45ce-b058-0a6b3d814c42 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.621046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.621682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.621959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.622230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] PATH_INFO: `/v3/projects/3938733b8d44494b94718d5196ad7c97/users/4f822c534b194488b2c012d78ee84292/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.638490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3938733b8d44494b94718d5196ad7c97, user_id=4f822c534b194488b2c012d78ee84292, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.641219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.655446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1065/2134] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:24 2022] PUT /identity/v3/projects/3938733b8d44494b94718d5196ad7c97/users/4f822c534b194488b2c012d78ee84292/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:24.659413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.670643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bb0092-1ee8-4ec3-9b09-979be728e4ac admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.730046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7d1dbb-e1ec-4135-a49e-5ea8677bc57b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.730560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7d1dbb-e1ec-4135-a49e-5ea8677bc57b admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.730836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7d1dbb-e1ec-4135-a49e-5ea8677bc57b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.731134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7d1dbb-e1ec-4135-a49e-5ea8677bc57b admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.731763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7d1dbb-e1ec-4135-a49e-5ea8677bc57b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.733366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7d1dbb-e1ec-4135-a49e-5ea8677bc57b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.743665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1070/2135] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:24 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:24.747891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.758865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1faf334-a62f-4151-b194-079da57b5267 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.763504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.764142 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.764418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.764694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] PATH_INFO: `/v3/projects/3938733b8d44494b94718d5196ad7c97/users/4f822c534b194488b2c012d78ee84292/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.766582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3938733b8d44494b94718d5196ad7c97, user_id=4f822c534b194488b2c012d78ee84292, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.769226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.779987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1066/2136] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:24 2022] PUT /identity/v3/projects/3938733b8d44494b94718d5196ad7c97/users/4f822c534b194488b2c012d78ee84292/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:24.791015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78313813-c83d-4a98-8f65-42a8bcda5f14 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.791302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78313813-c83d-4a98-8f65-42a8bcda5f14 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.791577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78313813-c83d-4a98-8f65-42a8bcda5f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.820522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-78313813-c83d-4a98-8f65-42a8bcda5f14 None None] MFA Rules not processed for user `4f822c534b194488b2c012d78ee84292`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:24.821743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-78313813-c83d-4a98-8f65-42a8bcda5f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:24.845655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:24.894154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1071/2137] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:24 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:24.905009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:24.906803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:24.970931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48f083f-f1f5-4e86-8685-99e6d86ee99a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:24.975837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8906a340-be3b-4433-80fa-d35e72881550 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:24.976450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906a340-be3b-4433-80fa-d35e72881550 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:24.976728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906a340-be3b-4433-80fa-d35e72881550 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:24.977000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906a340-be3b-4433-80fa-d35e72881550 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:24.986951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8906a340-be3b-4433-80fa-d35e72881550 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:24.988326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8906a340-be3b-4433-80fa-d35e72881550 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:24.998022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1067/2138] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:24 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:25.472105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78313813-c83d-4a98-8f65-42a8bcda5f14 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:25.542510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78313813-c83d-4a98-8f65-42a8bcda5f14 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:25.546864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:25.547512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:25.547788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:25.548061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] PATH_INFO: `/v3/users/0c214914259a416d9a163d6912434994` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:25.549170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c214914259a416d9a163d6912434994)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:25.551101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:25.606628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c214914259a416d9a163d6912434994'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:25.607367 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c214914259a416d9a163d6912434994'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:25.614484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c214914259a416d9a163d6912434994'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:25.615158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c214914259a416d9a163d6912434994'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:25.628101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c214914259a416d9a163d6912434994'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:25.641610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1072/2139] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:25 2022] DELETE /identity/v3/users/0c214914259a416d9a163d6912434994 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:25.646032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:25.646315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:25.647135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:25.668977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:25.669772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:25.743495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1068/2140] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:25.753112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:25.757808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:25.811928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:25.838530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc60e140-0a3e-4860-8aca-d67413ba5610 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:25.842465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:25.843101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:25.843377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:25.843647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:25.854119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:25.855684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:25.868596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1073/2141] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:26.158787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:26.160298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:26.172533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bc69e9-d9e8-4073-8fcc-3dd126c58ea4 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:26.177394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:26.178780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:26.179104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:26.179377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] PATH_INFO: `/v3/projects/ea39567800cb4c71aabb51a765f8bc18` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:26.181033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea39567800cb4c71aabb51a765f8bc18)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:26.183207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:26.203417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:26.249816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea39567800cb4c71aabb51a765f8bc18', 'request_id': 'req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.257218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea39567800cb4c71aabb51a765f8bc18', 'request_id': 'req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.264547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea39567800cb4c71aabb51a765f8bc18', 'request_id': 'req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.288059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] The token cache is being invalidate because project ea39567800cb4c71aabb51a765f8bc18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:26.288429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.290186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.292820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1069/2142] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:26 2022] DELETE /identity/v3/projects/ea39567800cb4c71aabb51a765f8bc18 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:34:26.297798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:26.308538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9abae4a-a980-4367-ae78-13c23341010b service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:26.387574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:26.388206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:26.388479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:26.388755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] PATH_INFO: `/v3/users/4f822c534b194488b2c012d78ee84292` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:26.389870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f822c534b194488b2c012d78ee84292)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:26.391813 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:26.440452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f822c534b194488b2c012d78ee84292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.441134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f822c534b194488b2c012d78ee84292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.448640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f822c534b194488b2c012d78ee84292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.449375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f822c534b194488b2c012d78ee84292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.469151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-2c939ba4-2d05-4fda-824a-7e13e2b16dac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f822c534b194488b2c012d78ee84292'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.483511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1074/2143] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:26 2022] DELETE /identity/v3/users/4f822c534b194488b2c012d78ee84292 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:26.778545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:26.779980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:26.793954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f33a233-c9b9-4e64-9d72-3c39f3b2bde7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:26.857899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:26.858480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:26.858754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:26.859061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] PATH_INFO: `/v3/projects/3938733b8d44494b94718d5196ad7c97` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:26.860178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] RBAC: Authorizing `identity:delete_project(project_id=3938733b8d44494b94718d5196ad7c97)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:26.862083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:26.911461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3938733b8d44494b94718d5196ad7c97', 'request_id': 'req-c2a913f4-16be-4f95-bc21-7192f130108f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.918902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3938733b8d44494b94718d5196ad7c97', 'request_id': 'req-c2a913f4-16be-4f95-bc21-7192f130108f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.924518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3938733b8d44494b94718d5196ad7c97', 'request_id': 'req-c2a913f4-16be-4f95-bc21-7192f130108f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.937572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] The token cache is being invalidate because project 3938733b8d44494b94718d5196ad7c97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:26.937852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.938443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c2a913f4-16be-4f95-bc21-7192f130108f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:26.939985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1070/2144] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:26 2022] DELETE /identity/v3/projects/3938733b8d44494b94718d5196ad7c97 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:26.945428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b628af5e-08df-46eb-b6bd-d1f234e8e3a9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:26.945750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b628af5e-08df-46eb-b6bd-d1f234e8e3a9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:26.946034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b628af5e-08df-46eb-b6bd-d1f234e8e3a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:26.966977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b628af5e-08df-46eb-b6bd-d1f234e8e3a9 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:26.968577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b628af5e-08df-46eb-b6bd-d1f234e8e3a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:27.032598 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1075/2145] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:27.041048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ce9026-904e-44ca-afd2-50601433a2ae None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.041361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ce9026-904e-44ca-afd2-50601433a2ae None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.041630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ce9026-904e-44ca-afd2-50601433a2ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.061079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-44ce9026-904e-44ca-afd2-50601433a2ae None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:27.062274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-44ce9026-904e-44ca-afd2-50601433a2ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:27.072220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1071/2146] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:27.076254 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b628af5e-08df-46eb-b6bd-d1f234e8e3a9 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:27.081460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:27.089082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b628af5e-08df-46eb-b6bd-d1f234e8e3a9 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:27.095179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:27.095784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.096066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.096347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.096984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:27.097286 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:27.099055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:27.113888 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1076/2147] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:27.115568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ce9026-904e-44ca-afd2-50601433a2ae None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:27.125037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ce9026-904e-44ca-afd2-50601433a2ae None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:27.129882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:27.130519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.131340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.131610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.132241 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:27.134111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:27.162833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1072/2148] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:27 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:27.167390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:27.177604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e88a2b-e0b8-4a53-8f27-f5dab68e1666 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:27.234292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:27.234894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.235202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.235478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.236118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:27.238045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:27.261580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1077/2149] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:27 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:27.266049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:27.277249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d147b0f3-4880-4bf6-9e4b-bafec600edb0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:27.281899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:27.282491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.282764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.283061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.283727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:27.285304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:27.286570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:27.297393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1073/2150] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:27.301089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:27.311597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b4e730-d897-4edb-b39c-f44a9c92b85c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:27.316437 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8db2618-92c5-4c90-afa9-a215470fffc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:27.317028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db2618-92c5-4c90-afa9-a215470fffc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.317309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db2618-92c5-4c90-afa9-a215470fffc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.317606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8db2618-92c5-4c90-afa9-a215470fffc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.318758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8db2618-92c5-4c90-afa9-a215470fffc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:27.320473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8db2618-92c5-4c90-afa9-a215470fffc2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:27.332104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1078/2151] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:27.337424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:27.347875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8d7e83-581e-48f2-8baf-2ef0083763cb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:27.353140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:27.353792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.354067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.354337 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] PATH_INFO: `/v3/projects/5c9e068e55504f5d80eae4683f1783cc/users/9d6bd20f633642f38a109857e832912f/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.370040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c9e068e55504f5d80eae4683f1783cc, user_id=9d6bd20f633642f38a109857e832912f, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:27.372669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:27.385765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1074/2152] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:27 2022] PUT /identity/v3/projects/5c9e068e55504f5d80eae4683f1783cc/users/9d6bd20f633642f38a109857e832912f/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:27.397402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a43b80-db00-4a70-b15f-ddab2abd8a53 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.397690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a43b80-db00-4a70-b15f-ddab2abd8a53 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.397968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a43b80-db00-4a70-b15f-ddab2abd8a53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.439336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-25a43b80-db00-4a70-b15f-ddab2abd8a53 None None] MFA Rules not processed for user `9d6bd20f633642f38a109857e832912f`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:27.440675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-25a43b80-db00-4a70-b15f-ddab2abd8a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:27.509655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1079/2153] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:27 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:27.522313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:27.524213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:27.584564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a55425-7720-4bc3-b13d-04aa91e7dff0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:27.588216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff05918b-a9ac-4459-b825-fb61c6449c16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:27.588873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05918b-a9ac-4459-b825-fb61c6449c16 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:27.589154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05918b-a9ac-4459-b825-fb61c6449c16 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:27.589426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05918b-a9ac-4459-b825-fb61c6449c16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:27.597623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05918b-a9ac-4459-b825-fb61c6449c16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:27.599115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05918b-a9ac-4459-b825-fb61c6449c16 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:27.609692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1075/2154] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:27 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:37.506057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=71,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:37.604361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=38,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:41.288134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a43b80-db00-4a70-b15f-ddab2abd8a53 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:41.366918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a43b80-db00-4a70-b15f-ddab2abd8a53 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:41.373078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:41.373745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:41.374027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:41.374362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] PATH_INFO: `/v3/users/9d6bd20f633642f38a109857e832912f` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:41.375672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d6bd20f633642f38a109857e832912f)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:41.377698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:41.433802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d6bd20f633642f38a109857e832912f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:41.434549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d6bd20f633642f38a109857e832912f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:41.441915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d6bd20f633642f38a109857e832912f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:41.442551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d6bd20f633642f38a109857e832912f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:41.454181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d6bd20f633642f38a109857e832912f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:41.468622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1080/2155] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:41 2022] DELETE /identity/v3/users/9d6bd20f633642f38a109857e832912f => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:41.473258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:41.473867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:41.474875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:41.495180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:41.496398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:41.561043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1076/2156] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:34:41.570651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:41.572182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:41.630610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa07c4c-ca2f-4986-a344-e157f51cc15e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:41.634228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416f4923-0890-42a4-9137-11327cee0c98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:41.634825 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416f4923-0890-42a4-9137-11327cee0c98 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:41.635137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416f4923-0890-42a4-9137-11327cee0c98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:41.635411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416f4923-0890-42a4-9137-11327cee0c98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:41.644258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416f4923-0890-42a4-9137-11327cee0c98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:41.645685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416f4923-0890-42a4-9137-11327cee0c98 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:41.655465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1081/2157] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:41.919180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:41.920674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:41.934479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a8aaaa-accd-411d-9b78-3778a9b05b5c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:41.939339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:41.939965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:41.940241 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:41.940511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] PATH_INFO: `/v3/projects/5c9e068e55504f5d80eae4683f1783cc` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:41.941618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c9e068e55504f5d80eae4683f1783cc)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:41.944189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:41.992636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c9e068e55504f5d80eae4683f1783cc', 'request_id': 'req-e08ab206-561d-4fd4-8078-9813f236ee44'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:41.999097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c9e068e55504f5d80eae4683f1783cc', 'request_id': 'req-e08ab206-561d-4fd4-8078-9813f236ee44'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.004535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c9e068e55504f5d80eae4683f1783cc', 'request_id': 'req-e08ab206-561d-4fd4-8078-9813f236ee44'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.016687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] The token cache is being invalidate because project 5c9e068e55504f5d80eae4683f1783cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:42.016975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.017590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e08ab206-561d-4fd4-8078-9813f236ee44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.019154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1077/2158] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:41 2022] DELETE /identity/v3/projects/5c9e068e55504f5d80eae4683f1783cc => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:42.024122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cdcbea-8a1e-4e5f-bffb-d4cf5f902433 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.024405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cdcbea-8a1e-4e5f-bffb-d4cf5f902433 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.024675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cdcbea-8a1e-4e5f-bffb-d4cf5f902433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.044921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-20cdcbea-8a1e-4e5f-bffb-d4cf5f902433 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:42.046096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-20cdcbea-8a1e-4e5f-bffb-d4cf5f902433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:42.063026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:42.108569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1082/2159] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:42.116658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5be9061-e68a-4834-be6a-c8ecf050246e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.116946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5be9061-e68a-4834-be6a-c8ecf050246e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.117223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5be9061-e68a-4834-be6a-c8ecf050246e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.136991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f5be9061-e68a-4834-be6a-c8ecf050246e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:42.138182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f5be9061-e68a-4834-be6a-c8ecf050246e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:42.146143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1078/2160] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:42.150508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cdcbea-8a1e-4e5f-bffb-d4cf5f902433 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.160047 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cdcbea-8a1e-4e5f-bffb-d4cf5f902433 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.165112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.165692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.165964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.166231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.166865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:42.167194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.168783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.180902 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1083/2161] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:42.185006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5be9061-e68a-4834-be6a-c8ecf050246e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.194470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5be9061-e68a-4834-be6a-c8ecf050246e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.199102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.199688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.199963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.200234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.200893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.202774 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.231415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1079/2162] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:42 2022] POST /identity/v3/projects => generated 361 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:42.235505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.246249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7efbcd-ad94-48bd-a390-e1c2f7943df9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.306009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.306608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.306880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.307186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.307822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.309730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.335409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1084/2163] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:42 2022] POST /identity/v3/users => generated 281 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:42.339578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.351894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60adf02c-43ae-4021-8f41-5cc970b1c42c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.356892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.357207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.357488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.357764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.358406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.360034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.373889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1080/2164] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:34:42.376957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.387276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e4bb13-4da3-49c1-83f4-59b266846adb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.391865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1086a4d4-9a78-4187-ad11-1dfed29758ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.392463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1086a4d4-9a78-4187-ad11-1dfed29758ed admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.392742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1086a4d4-9a78-4187-ad11-1dfed29758ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.393015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1086a4d4-9a78-4187-ad11-1dfed29758ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.393659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1086a4d4-9a78-4187-ad11-1dfed29758ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.395276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1086a4d4-9a78-4187-ad11-1dfed29758ed admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.406573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1085/2165] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:34:42.410748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.421785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f99460-bf81-48bc-8d55-1fc4f70f9b67 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.432231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.432878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.433151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.433418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] PATH_INFO: `/v3/projects/c0be7e71a9e74f59a2ecdb7499e42941/users/2c207888c1144af69c2757966bbb21d9/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.448151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0be7e71a9e74f59a2ecdb7499e42941, user_id=2c207888c1144af69c2757966bbb21d9, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.450760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.464062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1081/2166] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:42 2022] PUT /identity/v3/projects/c0be7e71a9e74f59a2ecdb7499e42941/users/2c207888c1144af69c2757966bbb21d9/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:42.474949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea233c4-dd8b-49f9-9278-10f487a6bec2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.475406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea233c4-dd8b-49f9-9278-10f487a6bec2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.475677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea233c4-dd8b-49f9-9278-10f487a6bec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.503002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-2ea233c4-dd8b-49f9-9278-10f487a6bec2 None None] MFA Rules not processed for user `2c207888c1144af69c2757966bbb21d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:42.504200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2ea233c4-dd8b-49f9-9278-10f487a6bec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:42.567015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1086/2167] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:42.602228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.603937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:42.626605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:42.667413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc25734-feb6-4a0b-85f0-d8b1754c3f3b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.671139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d823993f-ffb6-4532-98d4-d770f3224c7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.671751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d823993f-ffb6-4532-98d4-d770f3224c7f service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.672027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d823993f-ffb6-4532-98d4-d770f3224c7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.672298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d823993f-ffb6-4532-98d4-d770f3224c7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.681340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d823993f-ffb6-4532-98d4-d770f3224c7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.682782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d823993f-ffb6-4532-98d4-d770f3224c7f service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.693018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1082/2168] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:42.710007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea233c4-dd8b-49f9-9278-10f487a6bec2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:42.730652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:42.779405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea233c4-dd8b-49f9-9278-10f487a6bec2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:42.783946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:42.784580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.784851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.785123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] PATH_INFO: `/v3/users/2c207888c1144af69c2757966bbb21d9` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.786267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c207888c1144af69c2757966bbb21d9)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:42.788420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:42.838539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c207888c1144af69c2757966bbb21d9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.839227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c207888c1144af69c2757966bbb21d9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.847257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c207888c1144af69c2757966bbb21d9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.847982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c207888c1144af69c2757966bbb21d9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.861686 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c207888c1144af69c2757966bbb21d9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:42.879679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1087/2169] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:42 2022] DELETE /identity/v3/users/2c207888c1144af69c2757966bbb21d9 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:42.885470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:42.885796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:42.886030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:42.931898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:42.933121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:43.052702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1083/2170] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:34:43.059363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.060806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:43.132532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd082e-f281-411d-8e14-98e3682d3226 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.136100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89087101-958d-48ca-8388-39580dcc16e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.136700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89087101-958d-48ca-8388-39580dcc16e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.136974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89087101-958d-48ca-8388-39580dcc16e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.137243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89087101-958d-48ca-8388-39580dcc16e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.146329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89087101-958d-48ca-8388-39580dcc16e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.147791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89087101-958d-48ca-8388-39580dcc16e8 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.157570 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1088/2171] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:43.430725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.432407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:43.451617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035aeb0c-df24-49ce-a20a-df17af5a5726 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.456793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.457690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.458117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.458455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] PATH_INFO: `/v3/projects/c0be7e71a9e74f59a2ecdb7499e42941` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.459957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0be7e71a9e74f59a2ecdb7499e42941)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.462085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.511404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0be7e71a9e74f59a2ecdb7499e42941', 'request_id': 'req-37986523-3e0e-4c49-aecc-e4cb5c67a906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:43.517994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0be7e71a9e74f59a2ecdb7499e42941', 'request_id': 'req-37986523-3e0e-4c49-aecc-e4cb5c67a906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:43.527017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0be7e71a9e74f59a2ecdb7499e42941', 'request_id': 'req-37986523-3e0e-4c49-aecc-e4cb5c67a906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:43.540637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] The token cache is being invalidate because project c0be7e71a9e74f59a2ecdb7499e42941 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:43.540949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:43.541567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-37986523-3e0e-4c49-aecc-e4cb5c67a906 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:43.543068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1084/2172] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:43 2022] DELETE /identity/v3/projects/c0be7e71a9e74f59a2ecdb7499e42941 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:43.547801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8cc727-f317-497a-8fe5-450593cc2d42 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.548284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8cc727-f317-497a-8fe5-450593cc2d42 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.548386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8cc727-f317-497a-8fe5-450593cc2d42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.568344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3c8cc727-f317-497a-8fe5-450593cc2d42 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:43.569544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3c8cc727-f317-497a-8fe5-450593cc2d42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:43.604625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:43.634578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1089/2173] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:43.643338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce27acb-e481-4441-9345-1ff0731a0cca None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.643840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce27acb-e481-4441-9345-1ff0731a0cca None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.643937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce27acb-e481-4441-9345-1ff0731a0cca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.664159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6ce27acb-e481-4441-9345-1ff0731a0cca None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:43.665347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6ce27acb-e481-4441-9345-1ff0731a0cca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:43.673132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1085/2174] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:43.677242 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8cc727-f317-497a-8fe5-450593cc2d42 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.687084 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8cc727-f317-497a-8fe5-450593cc2d42 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.691714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.692308 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.692586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.693377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.694017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:34:43.694318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.695937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.708166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1090/2175] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:34:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:34:43.712562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce27acb-e481-4441-9345-1ff0731a0cca None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.721644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce27acb-e481-4441-9345-1ff0731a0cca None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.726904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.727513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.727786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.728053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.728678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.730542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.757530 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1086/2176] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:34:43 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:43.761817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.771832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fae57b-3933-4cdd-a0e0-dd06547f7117 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.827567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.828161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.828441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.828716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.829340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.831278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.860259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1091/2177] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:34:43 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:34:43.861478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.875054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70aacd5-0015-4489-a0a8-521aefc5db0a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.880248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.881540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.881818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.882085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.882724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.884345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.894397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1087/2178] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:43 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:43.898592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.908441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2d336-208f-4374-a3d2-01a1873d07f5 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.913072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4eb1ac-51bf-4d5f-a1c8-e0e13b2e2d74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.913651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4eb1ac-51bf-4d5f-a1c8-e0e13b2e2d74 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.913927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4eb1ac-51bf-4d5f-a1c8-e0e13b2e2d74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.914197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4eb1ac-51bf-4d5f-a1c8-e0e13b2e2d74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.914830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4eb1ac-51bf-4d5f-a1c8-e0e13b2e2d74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.916476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4eb1ac-51bf-4d5f-a1c8-e0e13b2e2d74 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.926793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1092/2179] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:34:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:34:43.931454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:43.941167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4329688-8cb7-4cb2-81f8-000e9ab54833 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:43.946429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:43.947093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.947373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.947645 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] PATH_INFO: `/v3/projects/785b4d979d8e463193899b424396a8b6/users/c57f3aedfe8148a49059a34bc396e5f9/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:43.949841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:43.965504 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=785b4d979d8e463193899b424396a8b6, user_id=c57f3aedfe8148a49059a34bc396e5f9, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:43.968263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:43.981177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1088/2180] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:34:43 2022] PUT /identity/v3/projects/785b4d979d8e463193899b424396a8b6/users/c57f3aedfe8148a49059a34bc396e5f9/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:43.992075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99931d2-9848-4885-b78b-ac4463f7d524 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:43.992357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99931d2-9848-4885-b78b-ac4463f7d524 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:43.992633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99931d2-9848-4885-b78b-ac4463f7d524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:44.023279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a99931d2-9848-4885-b78b-ac4463f7d524 None None] MFA Rules not processed for user `c57f3aedfe8148a49059a34bc396e5f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:44.024620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a99931d2-9848-4885-b78b-ac4463f7d524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:44.095403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1093/2181] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:44.108310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:44.110622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:44.190101 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41954330-caca-460e-9d12-308f7ccacac2 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:44.194122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eee228-c6e3-4ed2-bdec-883b0ebf0088 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:44.194716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eee228-c6e3-4ed2-bdec-883b0ebf0088 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:44.195024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eee228-c6e3-4ed2-bdec-883b0ebf0088 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:44.195304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eee228-c6e3-4ed2-bdec-883b0ebf0088 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:44.204914 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eee228-c6e3-4ed2-bdec-883b0ebf0088 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:44.206343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eee228-c6e3-4ed2-bdec-883b0ebf0088 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:44.217242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1089/2182] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:47.874161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99931d2-9848-4885-b78b-ac4463f7d524 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:47.912641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:47.937854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99931d2-9848-4885-b78b-ac4463f7d524 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:47.942278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:47.943588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:47.943880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:47.944147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] PATH_INFO: `/v3/users/c57f3aedfe8148a49059a34bc396e5f9` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:47.945251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c57f3aedfe8148a49059a34bc396e5f9)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:47.947170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:47.993413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c57f3aedfe8148a49059a34bc396e5f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:47.994066 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c57f3aedfe8148a49059a34bc396e5f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.000824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c57f3aedfe8148a49059a34bc396e5f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.001448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c57f3aedfe8148a49059a34bc396e5f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.013105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c57f3aedfe8148a49059a34bc396e5f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.026372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1094/2183] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:34:47 2022] DELETE /identity/v3/users/c57f3aedfe8148a49059a34bc396e5f9 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:48.030805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:48.031308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:48.031421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:48.051459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:48.052848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:48.118332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1090/2184] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:34:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:34:48.128255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:48.129634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:48.193993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534ba4c-1680-471f-bc0b-9a760c4421e8 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:48.197583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc68eb0-40ee-4771-95a7-8e41b154b518 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:48.198179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc68eb0-40ee-4771-95a7-8e41b154b518 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:48.198457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc68eb0-40ee-4771-95a7-8e41b154b518 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:48.198754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc68eb0-40ee-4771-95a7-8e41b154b518 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:48.208133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc68eb0-40ee-4771-95a7-8e41b154b518 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:48.210260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc68eb0-40ee-4771-95a7-8e41b154b518 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:48.223132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1095/2185] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:34:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:34:48.507061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:48.508586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:48.521319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea189ea9-c6b4-4bbc-967e-2e71c5b3e81b None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:48.526053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:48.526684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:48.526988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:48.527260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] PATH_INFO: `/v3/projects/785b4d979d8e463193899b424396a8b6` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:48.528354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=785b4d979d8e463193899b424396a8b6)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:48.530241 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:48.578366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '785b4d979d8e463193899b424396a8b6', 'request_id': 'req-a43ba44a-5256-44ad-a39f-2f2d51908c11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.585511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '785b4d979d8e463193899b424396a8b6', 'request_id': 'req-a43ba44a-5256-44ad-a39f-2f2d51908c11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.591054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '785b4d979d8e463193899b424396a8b6', 'request_id': 'req-a43ba44a-5256-44ad-a39f-2f2d51908c11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.604247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] The token cache is being invalidate because project 785b4d979d8e463193899b424396a8b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:34:48.604535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.605158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:34:48.606668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1091/2186] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:34:48 2022] DELETE /identity/v3/projects/785b4d979d8e463193899b424396a8b6 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:34:53.704608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3f3f74-8350-47d1-9c33-bfb942e38c0f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:53.705154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3f3f74-8350-47d1-9c33-bfb942e38c0f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:53.705662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3f3f74-8350-47d1-9c33-bfb942e38c0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:53.731346 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ec3f3f74-8350-47d1-9c33-bfb942e38c0f None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:34:53.732572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec3f3f74-8350-47d1-9c33-bfb942e38c0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:53.790128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1096/2187] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:34:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:34:53.831685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:34:53.833157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:34:53.843996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:34:53.904599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43ba44a-5256-44ad-a39f-2f2d51908c11 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:34:53.909943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:34:53.910729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:34:53.911234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:34:53.911850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:34:53.920032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:34:53.921625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:34:53.930226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1092/2188] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:34:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:35:03.783471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:03.926326 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=28 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:09.790074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:09.790592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:09.790861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:09.811006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:35:09.812204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:09.820043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1097/2189] 10.222.0.33 () {58 vars in 967 bytes} [Thu Oct 6 04:35:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:35:09.832954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:09.843522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab68a809-0f4e-4eeb-9d92-3872a1576cb3 service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:09.849234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:09.849833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:09.850114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:09.850392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:09.858507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:09.860128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:09.869421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1093/2190] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:35:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:35:16.048235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.049709 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:16.116401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e63a8f-7b82-4ecd-b47a-8a112b84371b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.121153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.121771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.122049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.122321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] PATH_INFO: `/v3/users/db187750f7514609923133b9a5f17ae7` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.123471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db187750f7514609923133b9a5f17ae7)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.125368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.170477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db187750f7514609923133b9a5f17ae7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.171161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db187750f7514609923133b9a5f17ae7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.177965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db187750f7514609923133b9a5f17ae7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.178589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db187750f7514609923133b9a5f17ae7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.190158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-51fe3f7e-17dc-467c-b077-480e1e7db8d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db187750f7514609923133b9a5f17ae7'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.202545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1098/2191] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:35:16 2022] DELETE /identity/v3/users/db187750f7514609923133b9a5f17ae7 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:35:16.377079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.389377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d377fa-7c66-4949-bf0f-0493c3f7404f service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.451304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.451925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.452186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.452455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] PATH_INFO: `/v3/projects/250b86943e2d45dc992421c2adee42b2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.453572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=250b86943e2d45dc992421c2adee42b2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.455534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.502405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '250b86943e2d45dc992421c2adee42b2', 'request_id': 'req-426f8263-5a49-485a-b586-586afeda35f5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.508663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '250b86943e2d45dc992421c2adee42b2', 'request_id': 'req-426f8263-5a49-485a-b586-586afeda35f5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.514314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '250b86943e2d45dc992421c2adee42b2', 'request_id': 'req-426f8263-5a49-485a-b586-586afeda35f5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.526463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] The token cache is being invalidate because project 250b86943e2d45dc992421c2adee42b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:35:16.526745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.527372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-426f8263-5a49-485a-b586-586afeda35f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:35:16.528878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1094/2192] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:35:16 2022] DELETE /identity/v3/projects/250b86943e2d45dc992421c2adee42b2 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:35:16.533950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff743faa-1981-4a3c-884d-27c98a782075 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.534263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff743faa-1981-4a3c-884d-27c98a782075 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.534541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff743faa-1981-4a3c-884d-27c98a782075 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.555971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ff743faa-1981-4a3c-884d-27c98a782075 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:35:16.557549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ff743faa-1981-4a3c-884d-27c98a782075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:16.616162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1099/2193] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:35:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:35:16.624025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c482c-44a2-4ca1-891a-021b6fc75846 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.624311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c482c-44a2-4ca1-891a-021b6fc75846 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.624589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4c482c-44a2-4ca1-891a-021b6fc75846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.644015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9f4c482c-44a2-4ca1-891a-021b6fc75846 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:35:16.645351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9f4c482c-44a2-4ca1-891a-021b6fc75846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:16.653187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1095/2194] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:35:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:35:16.657357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff743faa-1981-4a3c-884d-27c98a782075 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.667443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff743faa-1981-4a3c-884d-27c98a782075 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.671932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.672511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.672784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.673051 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.673682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:35:16.673983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.675605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.684705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:16.688275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1100/2195] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:35:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:35:16.692847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4c482c-44a2-4ca1-891a-021b6fc75846 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.702079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4c482c-44a2-4ca1-891a-021b6fc75846 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.706685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.707370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.707685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.707957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.708588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.710474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.739136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1096/2196] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:35:16 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:35:16.750174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.758437 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24aa24c3-03c1-4651-b3aa-534d1901b6e4 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.815945 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.816596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.816871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.817141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.817781 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.819728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.845481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1101/2197] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:35:16 2022] POST /identity/v3/users => generated 285 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:35:16.846916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.858475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3c7cec-de2f-4c40-afc2-f7700955470c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.864605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.865316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.865599 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.865870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.866511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.868379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.878828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1097/2198] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:35:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:35:16.883400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.893748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2bc0ad-7933-47a9-a932-bb2028644990 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.899312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a44e97-cc74-45eb-9a24-c54fd637c240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.899977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a44e97-cc74-45eb-9a24-c54fd637c240 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.900315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a44e97-cc74-45eb-9a24-c54fd637c240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.900653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a44e97-cc74-45eb-9a24-c54fd637c240 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.901353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a44e97-cc74-45eb-9a24-c54fd637c240 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.903098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a44e97-cc74-45eb-9a24-c54fd637c240 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.915396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1102/2199] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:35:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:35:16.919706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:16.931340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36a290-f8bc-4cdc-acc1-81abd6860a6b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:16.935903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:16.936548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.936818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.937117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] PATH_INFO: `/v3/projects/9d35d8995a784eef95ea145d9d307ca3/users/5acfc3cc148343dc93d32c37a4e1bf18/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:16.955468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d35d8995a784eef95ea145d9d307ca3, user_id=5acfc3cc148343dc93d32c37a4e1bf18, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:16.959917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:16.977035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1098/2200] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:35:16 2022] PUT /identity/v3/projects/9d35d8995a784eef95ea145d9d307ca3/users/5acfc3cc148343dc93d32c37a4e1bf18/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:35:16.987706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480f23ac-1408-4fdf-9192-a64830fe0eaa None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:16.987994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480f23ac-1408-4fdf-9192-a64830fe0eaa None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:16.988294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480f23ac-1408-4fdf-9192-a64830fe0eaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:17.017162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-480f23ac-1408-4fdf-9192-a64830fe0eaa None None] MFA Rules not processed for user `5acfc3cc148343dc93d32c37a4e1bf18`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:35:17.018307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-480f23ac-1408-4fdf-9192-a64830fe0eaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:17.082756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1103/2201] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:35:16 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:35:17.093856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:17.095552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:17.110620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:17.155025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffaa8bb-8e21-42b3-8916-05dcd2e5c482 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:17.158550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b34c61ebdb9147d5b3d66cbfa556f38e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:17.159237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:17.159514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:17.159783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:17.168502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:17.169905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:17.178492 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1099/2202] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:35:17 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:35:27.078618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=66,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:27.174598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=20 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:31.843858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558466d4-bd61-4210-a140-53b9e4be0311 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:31.844376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558466d4-bd61-4210-a140-53b9e4be0311 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:31.844528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558466d4-bd61-4210-a140-53b9e4be0311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:31.865224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-558466d4-bd61-4210-a140-53b9e4be0311 None None] MFA Rules not processed for user `a58cd041c02d46dcb8acf6adb11ee9de`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:35:31.866475 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-558466d4-bd61-4210-a140-53b9e4be0311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:31.925376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1104/2203] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:35:31 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:35:32.296525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:35:32.297916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:32.380833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256f5830-e721-48b8-bc57-7599ec40c692 service glance] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:35:32.386262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:35:32.386869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:32.387172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:32.387443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:32.389160 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:32.403080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:35:32.405105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:35:32.417766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1100/2204] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:35:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:35:41.918590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=25 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:42.412653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=32 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:35:47.793014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc232ce-1c51-4525-9842-0e60c95211c9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:35:47.793591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc232ce-1c51-4525-9842-0e60c95211c9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:35:47.793591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc232ce-1c51-4525-9842-0e60c95211c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:35:47.812616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-7cc232ce-1c51-4525-9842-0e60c95211c9 None None] MFA Rules not processed for user `1bd18015d8334028ada90e42558d7942`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:35:47.813830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7cc232ce-1c51-4525-9842-0e60c95211c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:35:47.869421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1105/2205] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:35:47 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:35:57.862446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=25 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:07.107414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:07.183051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a68f7-f6ce-40ad-b0dc-594f9dab0cdb service placement] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:07.187758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:07.188373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:07.188644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:07.188913 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] PATH_INFO: `/v3/users/5acfc3cc148343dc93d32c37a4e1bf18` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:07.190032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] RBAC: Authorizing `identity:delete_user(user_id=5acfc3cc148343dc93d32c37a4e1bf18)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:07.192013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:07.245471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5acfc3cc148343dc93d32c37a4e1bf18'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:07.246854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5acfc3cc148343dc93d32c37a4e1bf18'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:07.256647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5acfc3cc148343dc93d32c37a4e1bf18'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:07.284298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5acfc3cc148343dc93d32c37a4e1bf18'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:07.285257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5acfc3cc148343dc93d32c37a4e1bf18'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:07.307124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1101/2206] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:36:07 2022] DELETE /identity/v3/users/5acfc3cc148343dc93d32c37a4e1bf18 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:36:07.309191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:07.310069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:07.310348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:07.343800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:07.346195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:07.424815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1106/2207] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:07.435910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:07.437373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:07.514311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5770eae-b9d9-45b1-9612-55fb694fddef admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:07.518367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1605ac8b-b71d-4795-8177-c837f04f38a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:07.519144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1605ac8b-b71d-4795-8177-c837f04f38a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:07.519638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1605ac8b-b71d-4795-8177-c837f04f38a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:07.520005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1605ac8b-b71d-4795-8177-c837f04f38a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:07.531046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1605ac8b-b71d-4795-8177-c837f04f38a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:07.532770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1605ac8b-b71d-4795-8177-c837f04f38a8 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:07.550171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1102/2208] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:36:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:36:07.933996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:07.936350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:07.956850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79e822c-0be2-4ac2-b11b-7f75be16e2f3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:07.968796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:07.969948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:07.970227 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:07.970693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] PATH_INFO: `/v3/projects/9d35d8995a784eef95ea145d9d307ca3` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:07.974296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d35d8995a784eef95ea145d9d307ca3)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:07.976623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.079877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d35d8995a784eef95ea145d9d307ca3', 'request_id': 'req-282e84c3-dacb-45c7-91dc-a04daab7714b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:08.085495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d35d8995a784eef95ea145d9d307ca3', 'request_id': 'req-282e84c3-dacb-45c7-91dc-a04daab7714b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:08.095859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d35d8995a784eef95ea145d9d307ca3', 'request_id': 'req-282e84c3-dacb-45c7-91dc-a04daab7714b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:08.113749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] The token cache is being invalidate because project 9d35d8995a784eef95ea145d9d307ca3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:36:08.114421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:08.115421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-282e84c3-dacb-45c7-91dc-a04daab7714b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:08.118091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1107/2209] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:36:07 2022] DELETE /identity/v3/projects/9d35d8995a784eef95ea145d9d307ca3 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:36:08.125306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e634f0c5-d589-4768-bf2a-cdf3fc3b64bc None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.125306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e634f0c5-d589-4768-bf2a-cdf3fc3b64bc None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.125306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e634f0c5-d589-4768-bf2a-cdf3fc3b64bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.152628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e634f0c5-d589-4768-bf2a-cdf3fc3b64bc None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:08.153978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e634f0c5-d589-4768-bf2a-cdf3fc3b64bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:08.173113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:08.233809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1103/2210] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:08.241481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef9541-db37-4547-a72c-00b3defced18 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.241766 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef9541-db37-4547-a72c-00b3defced18 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.242040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef9541-db37-4547-a72c-00b3defced18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.263872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-feef9541-db37-4547-a72c-00b3defced18 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:08.265059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-feef9541-db37-4547-a72c-00b3defced18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:08.272792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1108/2211] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:08.277522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e634f0c5-d589-4768-bf2a-cdf3fc3b64bc None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:08.287283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e634f0c5-d589-4768-bf2a-cdf3fc3b64bc None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:08.293216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:08.294223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.294730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.295290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.296494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:36:08.297048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:08.300117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.315096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1104/2212] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:36:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:36:08.319054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef9541-db37-4547-a72c-00b3defced18 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:08.332363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef9541-db37-4547-a72c-00b3defced18 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:08.337666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:08.339106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.339294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.339564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.340195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:08.342113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.372633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1109/2213] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:36:08 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:36:08.376835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:08.387009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370ad981-fc98-46b7-aead-054b10aa415e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:08.474691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:08.475341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.475616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.475885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.476629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:08.478576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.504978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1105/2214] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:36:08 2022] POST /identity/v3/users => generated 287 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:36:08.506187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:08.517866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a37aed-e232-4016-8bdb-fdbfad9ec5cf admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:08.522551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:08.523169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.523448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.523719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.524386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:08.526574 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.541262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1110/2215] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:36:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:36:08.542483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:08.552839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f9b056-42a5-4e71-b1ce-5c00780972c0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:08.557538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab41dd81-85ee-46c0-bf48-a880e85c2556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:08.558120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab41dd81-85ee-46c0-bf48-a880e85c2556 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.558397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab41dd81-85ee-46c0-bf48-a880e85c2556 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.558665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab41dd81-85ee-46c0-bf48-a880e85c2556 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.559334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab41dd81-85ee-46c0-bf48-a880e85c2556 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:08.560913 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab41dd81-85ee-46c0-bf48-a880e85c2556 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.574999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1106/2216] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:36:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:36:08.576081 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:08.586510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5bd6a2-b950-4a4c-a317-ae6b3cff5552 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:08.591835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:08.592730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.593044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.593379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] PATH_INFO: `/v3/projects/f91922bb66cb4f3ba3e78969c3047d0b/users/700bf1fe3b96441d9aef0a6e95774c69/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.608377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f91922bb66cb4f3ba3e78969c3047d0b, user_id=700bf1fe3b96441d9aef0a6e95774c69, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:08.611073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.624544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1111/2217] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:36:08 2022] PUT /identity/v3/projects/f91922bb66cb4f3ba3e78969c3047d0b/users/700bf1fe3b96441d9aef0a6e95774c69/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:36:08.691745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8adf4d-d43a-4c50-af24-a21a27b314be None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.692026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8adf4d-d43a-4c50-af24-a21a27b314be None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.692297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8adf4d-d43a-4c50-af24-a21a27b314be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.724461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8e8adf4d-d43a-4c50-af24-a21a27b314be None None] MFA Rules not processed for user `700bf1fe3b96441d9aef0a6e95774c69`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:08.725727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8e8adf4d-d43a-4c50-af24-a21a27b314be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:08.792968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1107/2218] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:08.804025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:08.805832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:08.830422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:08.869942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da2f9b-0498-4392-9b07-a3c3ae684c3c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:08.873543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c24746-e13c-4d57-99df-68e1921c656c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:08.874141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c24746-e13c-4d57-99df-68e1921c656c service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:08.874418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c24746-e13c-4d57-99df-68e1921c656c service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:08.874690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c24746-e13c-4d57-99df-68e1921c656c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:08.882764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c24746-e13c-4d57-99df-68e1921c656c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:08.884225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c24746-e13c-4d57-99df-68e1921c656c service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:08.893484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1112/2219] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:36:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:36:10.267751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:10.268237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:10.268237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:10.287881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] MFA Rules not processed for user `700bf1fe3b96441d9aef0a6e95774c69`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:10.289112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:10.296561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1108/2220] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:18.889618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=17 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:20.282342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:40.653708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e733d6c9-b7f9-4b81-b84c-1f0ea8ba0923 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:40.654238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e733d6c9-b7f9-4b81-b84c-1f0ea8ba0923 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:40.654341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e733d6c9-b7f9-4b81-b84c-1f0ea8ba0923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:40.674099 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e733d6c9-b7f9-4b81-b84c-1f0ea8ba0923 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:40.675319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e733d6c9-b7f9-4b81-b84c-1f0ea8ba0923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:40.727893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1113/2221] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:36:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:40.775707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:40.777113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:40.844364 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8fa6cb-4c81-4936-9c43-5cb74f8b265d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:40.849505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:40.850107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:40.850385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:40.850655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:40.859339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:40.860894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:40.869210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1109/2222] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:36:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:36:41.369970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ba1df-c8a3-4e42-9792-abb84f2b3604 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:41.370256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ba1df-c8a3-4e42-9792-abb84f2b3604 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:41.370626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ba1df-c8a3-4e42-9792-abb84f2b3604 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:41.391808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f75ba1df-c8a3-4e42-9792-abb84f2b3604 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:41.392926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f75ba1df-c8a3-4e42-9792-abb84f2b3604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:41.399877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1114/2223] 10.222.0.33 () {58 vars in 967 bytes} [Thu Oct 6 04:36:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:41.412526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:41.423861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9927e8a-ecee-4da5-a272-3f88caa1a23b service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:41.429176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:41.429769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:41.430044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:41.430468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:41.439689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:41.441251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:41.449321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1110/2224] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:36:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:36:41.671990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310525a9-d95b-49b4-a27b-59c71c1bca10 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:41.672483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310525a9-d95b-49b4-a27b-59c71c1bca10 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:41.672580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310525a9-d95b-49b4-a27b-59c71c1bca10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:41.691339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-310525a9-d95b-49b4-a27b-59c71c1bca10 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:41.692507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-310525a9-d95b-49b4-a27b-59c71c1bca10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:41.699408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1115/2225] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:36:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:41.711722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:41.722642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea13835-bf42-48e3-b5b4-92c102fb3494 service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:41.727752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:41.728338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:41.728613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:41.728888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:41.736724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:41.738271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:41.747443 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1111/2226] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:36:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:36:42.074678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d62f98-992b-4109-ad73-35ae51e5ce4f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:42.075621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d62f98-992b-4109-ad73-35ae51e5ce4f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:42.075770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d62f98-992b-4109-ad73-35ae51e5ce4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:42.095044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-34d62f98-992b-4109-ad73-35ae51e5ce4f None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:42.096244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-34d62f98-992b-4109-ad73-35ae51e5ce4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:42.103388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1116/2227] 10.222.0.33 () {58 vars in 967 bytes} [Thu Oct 6 04:36:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:42.116117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:42.128104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b85502-226e-478f-8f43-b68169f77933 service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:42.133282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:42.133868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:42.134146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:42.134412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:42.144710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:42.146275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:42.155539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1112/2228] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:36:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:36:43.727048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0ecd16-f2ca-4de9-9427-772d3b9f3607 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:43.727534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0ecd16-f2ca-4de9-9427-772d3b9f3607 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:43.727743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0ecd16-f2ca-4de9-9427-772d3b9f3607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:43.748313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5b0ecd16-f2ca-4de9-9427-772d3b9f3607 None None] MFA Rules not processed for user `f68d73b5efc74b3380429cf72e506905`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:43.749566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5b0ecd16-f2ca-4de9-9427-772d3b9f3607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:43.804561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1117/2229] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:44.446279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:44.521634 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a46f7bb-8712-4f67-ba39-9c67e66c2841 service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:44.527356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:44.528887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:44.529185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:44.529465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] PATH_INFO: `/v3/users/700bf1fe3b96441d9aef0a6e95774c69` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:44.530991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] RBAC: Authorizing `identity:delete_user(user_id=700bf1fe3b96441d9aef0a6e95774c69)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:44.532538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:44.582887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '700bf1fe3b96441d9aef0a6e95774c69'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:44.583562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '700bf1fe3b96441d9aef0a6e95774c69'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:44.588069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:44.591248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '700bf1fe3b96441d9aef0a6e95774c69'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:44.606628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '700bf1fe3b96441d9aef0a6e95774c69'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:44.607452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '700bf1fe3b96441d9aef0a6e95774c69'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:44.620791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1113/2230] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:36:44 2022] DELETE /identity/v3/users/700bf1fe3b96441d9aef0a6e95774c69 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:36:44.625522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:44.625814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:44.626091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:44.649184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:44.650660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:44.726831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1118/2231] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:44.737391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:44.738899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:44.798659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572304ac-6e7c-4ab6-b448-251b2312d68f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:44.802355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687f9c5d-0483-4ad2-aacb-d0706713a610 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:44.802950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f9c5d-0483-4ad2-aacb-d0706713a610 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:44.803252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f9c5d-0483-4ad2-aacb-d0706713a610 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:44.803522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687f9c5d-0483-4ad2-aacb-d0706713a610 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:44.813663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f9c5d-0483-4ad2-aacb-d0706713a610 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:44.815123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687f9c5d-0483-4ad2-aacb-d0706713a610 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:44.825167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1114/2232] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:36:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:36:44.998641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.000373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:45.025388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85c91ef-76f5-462e-bd35-6c9ab2bbfaf7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.031846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.032839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.033237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.033631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] PATH_INFO: `/v3/projects/f91922bb66cb4f3ba3e78969c3047d0b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.034968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f91922bb66cb4f3ba3e78969c3047d0b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.037145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.070625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:45.088870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f91922bb66cb4f3ba3e78969c3047d0b', 'request_id': 'req-96d199fe-36f1-41a6-86ac-1004daff0204'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:45.095534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f91922bb66cb4f3ba3e78969c3047d0b', 'request_id': 'req-96d199fe-36f1-41a6-86ac-1004daff0204'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:45.101816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f91922bb66cb4f3ba3e78969c3047d0b', 'request_id': 'req-96d199fe-36f1-41a6-86ac-1004daff0204'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:45.115309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] The token cache is being invalidate because project f91922bb66cb4f3ba3e78969c3047d0b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:36:45.115596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:45.116203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-96d199fe-36f1-41a6-86ac-1004daff0204 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:45.117700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1119/2233] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:36:44 2022] DELETE /identity/v3/projects/f91922bb66cb4f3ba3e78969c3047d0b => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:36:45.122901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3de2b9-34e9-4cfe-9a5f-fb6fee5dfa28 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.123289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3de2b9-34e9-4cfe-9a5f-fb6fee5dfa28 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.123639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3de2b9-34e9-4cfe-9a5f-fb6fee5dfa28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.145410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ce3de2b9-34e9-4cfe-9a5f-fb6fee5dfa28 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:45.146986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ce3de2b9-34e9-4cfe-9a5f-fb6fee5dfa28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:45.210011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1115/2234] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:45.217697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9600c2-44c3-4bac-9161-8e10a3be9245 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.217980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9600c2-44c3-4bac-9161-8e10a3be9245 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.218250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9600c2-44c3-4bac-9161-8e10a3be9245 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.239403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fd9600c2-44c3-4bac-9161-8e10a3be9245 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:45.240593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fd9600c2-44c3-4bac-9161-8e10a3be9245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:45.249321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1120/2235] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:45.253701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3de2b9-34e9-4cfe-9a5f-fb6fee5dfa28 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.264175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3de2b9-34e9-4cfe-9a5f-fb6fee5dfa28 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.268812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.269433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.269735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.270009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.270679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:36:45.271002 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.272648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.289090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1116/2236] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:36:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:36:45.290147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9600c2-44c3-4bac-9161-8e10a3be9245 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.299844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9600c2-44c3-4bac-9161-8e10a3be9245 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.304452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.305037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.305314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.305586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.306208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.308403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.337350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1121/2237] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:36:45 2022] POST /identity/v3/projects => generated 399 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:36:45.341677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.352162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4715f8d4-87f8-4b45-9159-6875ced8a602 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.409924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.410521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.410798 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.411105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.411746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.413741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.421752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:45.440604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1117/2238] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:36:45 2022] POST /identity/v3/users => generated 300 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:36:45.441972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.453060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347a8c25-6f66-4e26-9696-c09fcee0e63d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.457633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.458228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.458501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.458798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.459466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.461046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.471623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1122/2239] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:36:45.481008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.486256 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fad376-79ee-4adb-b88f-8f301d654175 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.491108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d40fc7-f614-472c-b8a0-3523a4de4b61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.491696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d40fc7-f614-472c-b8a0-3523a4de4b61 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.491974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d40fc7-f614-472c-b8a0-3523a4de4b61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.492249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d40fc7-f614-472c-b8a0-3523a4de4b61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.492886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d40fc7-f614-472c-b8a0-3523a4de4b61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.494712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d40fc7-f614-472c-b8a0-3523a4de4b61 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.507412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1118/2240] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:36:45.511427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.522350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8e2377-2b0c-4a56-9501-a16fb3db95ab admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.526918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.528311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.528603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.528884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] PATH_INFO: `/v3/projects/77b4be786dd9412db00c6ba25c133a2b/users/bc4a6051d1c6441fac2c1b2e20316f7f/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.544660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77b4be786dd9412db00c6ba25c133a2b, user_id=bc4a6051d1c6441fac2c1b2e20316f7f, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.547376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.561625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1123/2241] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:36:45 2022] PUT /identity/v3/projects/77b4be786dd9412db00c6ba25c133a2b/users/bc4a6051d1c6441fac2c1b2e20316f7f/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:36:45.571476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7501e-4ff0-41ab-9a0d-945a9d87403d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.571796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7501e-4ff0-41ab-9a0d-945a9d87403d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.572074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7501e-4ff0-41ab-9a0d-945a9d87403d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.602214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ede7501e-4ff0-41ab-9a0d-945a9d87403d None None] MFA Rules not processed for user `bc4a6051d1c6441fac2c1b2e20316f7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:45.603471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ede7501e-4ff0-41ab-9a0d-945a9d87403d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:45.672327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1119/2242] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:45 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:45.683743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:45.685849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:45.754204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defe5feb-ef02-4ba2-afb5-4178223a71b6 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:45.758283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b242959-7e99-4537-880e-b5e3e910e274 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:45.759309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b242959-7e99-4537-880e-b5e3e910e274 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:45.759663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b242959-7e99-4537-880e-b5e3e910e274 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:45.760133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b242959-7e99-4537-880e-b5e3e910e274 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:45.778917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b242959-7e99-4537-880e-b5e3e910e274 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:45.780614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b242959-7e99-4537-880e-b5e3e910e274 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:45.790660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1124/2243] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:36:45 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:36:54.954057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede7501e-4ff0-41ab-9a0d-945a9d87403d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:55.036986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede7501e-4ff0-41ab-9a0d-945a9d87403d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:55.041758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:55.042393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:55.042669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:55.042938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] PATH_INFO: `/v3/users/bc4a6051d1c6441fac2c1b2e20316f7f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:55.044737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc4a6051d1c6441fac2c1b2e20316f7f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:55.046627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:55.095063 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc4a6051d1c6441fac2c1b2e20316f7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.095704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc4a6051d1c6441fac2c1b2e20316f7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.102398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc4a6051d1c6441fac2c1b2e20316f7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.113189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:INSERT=3,keystone:SELECT=89,keystone:DELETE=8 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:55.115752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc4a6051d1c6441fac2c1b2e20316f7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.116501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc4a6051d1c6441fac2c1b2e20316f7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.130755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1120/2244] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:36:54 2022] DELETE /identity/v3/users/bc4a6051d1c6441fac2c1b2e20316f7f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:36:55.135224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:55.135708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:55.135804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:55.156589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:36:55.157771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:55.203956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:36:55.223008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1125/2245] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:36:55.232823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:55.234238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:55.296588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a56056e-5eb7-47fc-ba00-b7292678c796 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:55.303963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:55.304611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:55.304822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:55.305092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:55.325668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:55.327324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:55.337359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1121/2246] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:36:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:36:55.615389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:36:55.616842 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:36:55.630107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5946c387-1415-49af-b69d-51055c3c13dc None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:36:55.635008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:36:55.635642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:36:55.635938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:36:55.636183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] PATH_INFO: `/v3/projects/77b4be786dd9412db00c6ba25c133a2b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:36:55.637456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77b4be786dd9412db00c6ba25c133a2b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:36:55.639650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:36:55.686641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77b4be786dd9412db00c6ba25c133a2b', 'request_id': 'req-4b8b206b-ad9e-4835-b960-1eb2775b0f07'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.692045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77b4be786dd9412db00c6ba25c133a2b', 'request_id': 'req-4b8b206b-ad9e-4835-b960-1eb2775b0f07'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.698291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77b4be786dd9412db00c6ba25c133a2b', 'request_id': 'req-4b8b206b-ad9e-4835-b960-1eb2775b0f07'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.710868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] The token cache is being invalidate because project 77b4be786dd9412db00c6ba25c133a2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:36:55.711178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.712284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:36:55.713785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1126/2247] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:36:55 2022] DELETE /identity/v3/projects/77b4be786dd9412db00c6ba25c133a2b => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:37:05.332849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=2,keystone:SELECT=29,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:05.707897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=30,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:08.983472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:08.984960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:08.996246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09750ef-e817-442e-b11b-3e27e0f24fea service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:09.047843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:09.048432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:09.048702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:09.048969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:09.050817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:09.121230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:09.122757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:09.134338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1122/2248] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:10.416528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:10.418006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:10.432589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8b206b-ad9e-4835-b960-1eb2775b0f07 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:10.436278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:10.436951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:10.437173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:10.437449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:10.439639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:10.456724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:10.458162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:10.470290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1127/2249] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:19.129533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=54 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:20.464287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:27.044766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:27.046128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:27.060274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01446519-cd1f-4552-b7cd-7211a2030a18 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:27.066314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:27.066896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:27.067204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:27.067482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:27.069151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:27.127626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:27.129254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:27.139120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1123/2250] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:27 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:29.599967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:29.602363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:29.681601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8011ac3-92d9-4de6-9e94-020ac542440d service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:29.690465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:29.691141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:29.691416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:29.691683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] PATH_INFO: `/v3/users/1bd18015d8334028ada90e42558d7942` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:29.692823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1bd18015d8334028ada90e42558d7942)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:29.694867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:29.741571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1bd18015d8334028ada90e42558d7942'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:29.742297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1bd18015d8334028ada90e42558d7942'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:29.749547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1bd18015d8334028ada90e42558d7942'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:29.750691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1bd18015d8334028ada90e42558d7942'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:29.762276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-66cd558a-c3cc-465a-8a0b-3c42fb6f4e76 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1bd18015d8334028ada90e42558d7942'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:29.775442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1128/2251] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:37:29 2022] DELETE /identity/v3/users/1bd18015d8334028ada90e42558d7942 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:37:29.946629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:29.957419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80394ea-d2a3-4ef7-b735-4b5e39a052ed service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.014864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.015511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.015797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.016536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] PATH_INFO: `/v3/projects/d951f19622f24014b0db552eafd4346d` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.017635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d951f19622f24014b0db552eafd4346d)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.019576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.064577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd951f19622f24014b0db552eafd4346d', 'request_id': 'req-a4d886e0-576d-4e32-a2ed-02947579e0f4'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:30.070878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd951f19622f24014b0db552eafd4346d', 'request_id': 'req-a4d886e0-576d-4e32-a2ed-02947579e0f4'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:30.076433 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd951f19622f24014b0db552eafd4346d', 'request_id': 'req-a4d886e0-576d-4e32-a2ed-02947579e0f4'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:30.089392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] The token cache is being invalidate because project d951f19622f24014b0db552eafd4346d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:37:30.089674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:30.090294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a4d886e0-576d-4e32-a2ed-02947579e0f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:37:30.091791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1124/2252] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:37:29 2022] DELETE /identity/v3/projects/d951f19622f24014b0db552eafd4346d => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:37:30.098231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c24027-25fb-4436-9b51-22dc2619392b None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.098518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c24027-25fb-4436-9b51-22dc2619392b None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.098795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c24027-25fb-4436-9b51-22dc2619392b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.118269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-73c24027-25fb-4436-9b51-22dc2619392b None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:30.119767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-73c24027-25fb-4436-9b51-22dc2619392b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:30.177679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1129/2253] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:37:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:30.185490 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbd924-57fb-4e47-8f32-0f57de3449dd None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.185781 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbd924-57fb-4e47-8f32-0f57de3449dd None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.186056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbd924-57fb-4e47-8f32-0f57de3449dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.206138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-24cbd924-57fb-4e47-8f32-0f57de3449dd None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:30.207290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-24cbd924-57fb-4e47-8f32-0f57de3449dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:30.215182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1125/2254] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:37:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:30.219940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c24027-25fb-4436-9b51-22dc2619392b None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:30.229801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c24027-25fb-4436-9b51-22dc2619392b None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.234428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.235138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.235417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.235689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.236330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:37:30.236629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.238260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.250238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1130/2255] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:37:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:37:30.254484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbd924-57fb-4e47-8f32-0f57de3449dd None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:30.263567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbd924-57fb-4e47-8f32-0f57de3449dd None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.268830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.269410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.269681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.269950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.270565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.272505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.294236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:30.301886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1126/2256] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:37:30 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:37:30.306323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:30.317656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8418c009-6ba5-4184-9988-1156f8e59cb3 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.321339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:30.374465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.375189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.375586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.375999 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.376625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.378539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.399566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1131/2257] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:37:30 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:37:30.403814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:30.414869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd47925c-7240-4aed-bf3a-43a62f2f4bcc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.419550 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.420134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.420410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.420678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.421320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.423224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.432948 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1127/2258] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:37:30.436914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:30.447512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dbf934-2330-4a47-a4d5-1059fe0ad088 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.452093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5612d4ec-7351-4089-bfe7-c39bfa98a7ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.452673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5612d4ec-7351-4089-bfe7-c39bfa98a7ca admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.452950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5612d4ec-7351-4089-bfe7-c39bfa98a7ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.453219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5612d4ec-7351-4089-bfe7-c39bfa98a7ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.454597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5612d4ec-7351-4089-bfe7-c39bfa98a7ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.456279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5612d4ec-7351-4089-bfe7-c39bfa98a7ca admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.466618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1132/2259] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:37:30.471062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:30.480915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54f0167-1f16-4959-ba19-6b9cae1eee4f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.485906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.486465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.486739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.487041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] PATH_INFO: `/v3/projects/43ee2f7c65044158a0230e4997d7bdf9/users/ddfa371a39b84f42bba4621ae3a99794/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.502233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] RBAC: Authorizing `identity:create_grant(project_id=43ee2f7c65044158a0230e4997d7bdf9, user_id=ddfa371a39b84f42bba4621ae3a99794, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.504894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.517832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1128/2260] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:37:30 2022] PUT /identity/v3/projects/43ee2f7c65044158a0230e4997d7bdf9/users/ddfa371a39b84f42bba4621ae3a99794/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:37:30.522851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5003e1a0-7c57-4b78-9e01-a719f5f6eae0 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.523168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5003e1a0-7c57-4b78-9e01-a719f5f6eae0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.523446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5003e1a0-7c57-4b78-9e01-a719f5f6eae0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.542287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5003e1a0-7c57-4b78-9e01-a719f5f6eae0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:30.543498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5003e1a0-7c57-4b78-9e01-a719f5f6eae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:30.601539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1133/2261] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:37:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:30.611130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:30.612829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:30.669576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53829e83-863b-4bbf-bed0-b2d595bb6deb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:30.673535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:30.674133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:30.674413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:30.674791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:30.683851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:30.685300 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:30.696804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1129/2262] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:32.845720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23257341-9bf5-4171-ba4d-c33d8665ed89 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:32.847444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23257341-9bf5-4171-ba4d-c33d8665ed89 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:32.848034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23257341-9bf5-4171-ba4d-c33d8665ed89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:32.871314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-23257341-9bf5-4171-ba4d-c33d8665ed89 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:32.872538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-23257341-9bf5-4171-ba4d-c33d8665ed89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:32.942709 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1134/2263] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:37:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:32.956871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:32.958333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:33.065112 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8977e9-4cfa-4bf9-a6f6-af9cbeddead0 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:33.075026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:33.075752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:33.076035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:33.076311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:33.089627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:33.091635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:33.101300 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1130/2264] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:37:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:37:35.448333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da2ece8-6eb6-4fd7-8bc0-086d8a3be535 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:35.449384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da2ece8-6eb6-4fd7-8bc0-086d8a3be535 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:35.449796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da2ece8-6eb6-4fd7-8bc0-086d8a3be535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:35.483918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-7da2ece8-6eb6-4fd7-8bc0-086d8a3be535 None None] MFA Rules not processed for user `ddfa371a39b84f42bba4621ae3a99794`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:35.485977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7da2ece8-6eb6-4fd7-8bc0-086d8a3be535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:35.520040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:35.602268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1135/2265] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:37:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:35.607022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:35.624132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7926ea8-35be-4946-b885-d7ce5c1669d9 service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:35.631061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:35.631682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:35.631962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:35.632236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:35.632886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:35.635102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:35.691624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1131/2266] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:37:35 2022] POST /identity/v3/projects => generated 385 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:37:35.691933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da2ece8-6eb6-4fd7-8bc0-086d8a3be535 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:35.706556 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da2ece8-6eb6-4fd7-8bc0-086d8a3be535 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:35.796528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:35.798804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:35.799360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:35.799845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:35.800628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:35.803042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:35.839816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1136/2267] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:37:35 2022] POST /identity/v3/users => generated 299 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:37:35.840620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:35.853229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebeb5df-6391-41e0-b31c-e4aeb38b5eeb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:35.859896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:35.860512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:35.860785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:35.861055 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:35.861698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:35.863359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:35.878290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1132/2268] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:37:35 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:37:35.882504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:35.894623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa314923-51cb-4293-a453-a71dbf81b7f7 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:35.900339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:35.901235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:35.902532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:35.903172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] PATH_INFO: `/v3/projects/d2b7266e22c2478e8e8b75229aae303e/users/9ebdc312cb65479ead36d13eef84ec5c/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:35.921565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d2b7266e22c2478e8e8b75229aae303e, user_id=9ebdc312cb65479ead36d13eef84ec5c, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:35.925264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:35.945542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1137/2269] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:37:35 2022] PUT /identity/v3/projects/d2b7266e22c2478e8e8b75229aae303e/users/9ebdc312cb65479ead36d13eef84ec5c/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:37:35.949481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:35.961894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa5d0cd-eb5f-4e65-93d7-f2607e1bf6f5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:35.989554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:36.039363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32271d32-4ef2-4fc3-99e2-b5003a46bcde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:36.040080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32271d32-4ef2-4fc3-99e2-b5003a46bcde admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:36.042606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32271d32-4ef2-4fc3-99e2-b5003a46bcde admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:36.043023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32271d32-4ef2-4fc3-99e2-b5003a46bcde admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:36.043785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32271d32-4ef2-4fc3-99e2-b5003a46bcde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:36.048121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32271d32-4ef2-4fc3-99e2-b5003a46bcde admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:36.061342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1133/2270] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:37:35 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:37:36.066156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:36.083992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26207f95-d0c2-4754-9162-640b506c5190 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:36.091321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:36.091948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:36.092216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:36.092538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] PATH_INFO: `/v3/projects/d2b7266e22c2478e8e8b75229aae303e/users/9ebdc312cb65479ead36d13eef84ec5c/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:36.095211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d2b7266e22c2478e8e8b75229aae303e, user_id=9ebdc312cb65479ead36d13eef84ec5c, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:36.098045 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:36.117819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1138/2271] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:37:36 2022] PUT /identity/v3/projects/d2b7266e22c2478e8e8b75229aae303e/users/9ebdc312cb65479ead36d13eef84ec5c/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:37:42.377552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a5969-94cc-4ea5-aa39-c912b941cb9f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:42.378481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a5969-94cc-4ea5-aa39-c912b941cb9f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:42.378909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a5969-94cc-4ea5-aa39-c912b941cb9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:42.417655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-441a5969-94cc-4ea5-aa39-c912b941cb9f None None] MFA Rules not processed for user `9ebdc312cb65479ead36d13eef84ec5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:42.419120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-441a5969-94cc-4ea5-aa39-c912b941cb9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:42.556529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1134/2272] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:37:42 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:37:42.573515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:42.575855 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:42.655936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e54025-d3bb-483e-bd10-6e109a999a2e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:42.661187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b34c61ebdb9147d5b3d66cbfa556f38e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:42.661852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:42.662073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:42.662348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:42.708240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:42.773634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:42.775456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:42.790253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1139/2273] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:42 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:42.993920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99506e8-600d-4a18-b707-6233a7da8ea8 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:42.994417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99506e8-600d-4a18-b707-6233a7da8ea8 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:42.994738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99506e8-600d-4a18-b707-6233a7da8ea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:43.017653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a99506e8-600d-4a18-b707-6233a7da8ea8 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:43.022929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a99506e8-600d-4a18-b707-6233a7da8ea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:43.123605 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1135/2274] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:37:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:37:43.139125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:43.245183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843e7ada-88f3-4a6c-ad87-06682d232b07 service glance] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:43.252638 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:43.253420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:43.253794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:43.254158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:43.267563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:43.269808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:43.283086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1140/2275] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:37:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:37:48.011696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99506e8-600d-4a18-b707-6233a7da8ea8 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:48.096806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99506e8-600d-4a18-b707-6233a7da8ea8 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:48.101404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6e66ab-eea6-4db0-a317-129a871cbfd4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:48.102382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e66ab-eea6-4db0-a317-129a871cbfd4 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:48.102907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e66ab-eea6-4db0-a317-129a871cbfd4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:48.103430 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6e66ab-eea6-4db0-a317-129a871cbfd4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:48.105938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9b6e66ab-eea6-4db0-a317-129a871cbfd4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:48.118413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:48.123865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6e66ab-eea6-4db0-a317-129a871cbfd4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:48.127835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6e66ab-eea6-4db0-a317-129a871cbfd4 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:48.139528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1136/2276] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:51.866984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:51.868416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:51.881559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2787b100-afcb-4eda-b69c-38292e05b4b5 service swift] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:51.885295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d397dc-ff92-49cf-bbdc-c42a61c2fe06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:51.885894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d397dc-ff92-49cf-bbdc-c42a61c2fe06 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:51.886171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d397dc-ff92-49cf-bbdc-c42a61c2fe06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:51.886462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d397dc-ff92-49cf-bbdc-c42a61c2fe06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:51.889959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-73d397dc-ff92-49cf-bbdc-c42a61c2fe06 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:51.901526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d397dc-ff92-49cf-bbdc-c42a61c2fe06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:51.902940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d397dc-ff92-49cf-bbdc-c42a61c2fe06 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:51.912943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1141/2277] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:52.080668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d766d-32c4-4249-974a-7a8ab716de36 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.081269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d766d-32c4-4249-974a-7a8ab716de36 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.081269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d766d-32c4-4249-974a-7a8ab716de36 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.082923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1137/2278] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:37:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:37:52.090154 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cdff8c-cfaa-48b3-bcc7-6379d2ed322c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.090436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cdff8c-cfaa-48b3-bcc7-6379d2ed322c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.090719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cdff8c-cfaa-48b3-bcc7-6379d2ed322c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.116646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-51cdff8c-cfaa-48b3-bcc7-6379d2ed322c None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:52.120557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-51cdff8c-cfaa-48b3-bcc7-6379d2ed322c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:52.223792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1142/2279] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:37:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:52.239362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d766d-32c4-4249-974a-7a8ab716de36 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:52.243106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ca1d766d-32c4-4249-974a-7a8ab716de36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:52.258593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d766d-32c4-4249-974a-7a8ab716de36 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:52.264526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf25d7d-e9c1-4aa0-aca0-a455815fc7a4 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b34c61ebdb9147d5b3d66cbfa556f38e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:52.265397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf25d7d-e9c1-4aa0-aca0-a455815fc7a4 service glance] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.265898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf25d7d-e9c1-4aa0-aca0-a455815fc7a4 service glance] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.266332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf25d7d-e9c1-4aa0-aca0-a455815fc7a4 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.280394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf25d7d-e9c1-4aa0-aca0-a455815fc7a4 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:52.282735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf25d7d-e9c1-4aa0-aca0-a455815fc7a4 service glance] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:52.295091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1138/2280] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:52.369090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81411cbd-93e9-4ee1-814c-9bff383f977c None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.369684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81411cbd-93e9-4ee1-814c-9bff383f977c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.369955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81411cbd-93e9-4ee1-814c-9bff383f977c None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.371379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1143/2281] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:37:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:37:52.377906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3792dbe5-2566-4b38-8cb3-9938b33c1401 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.378602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3792dbe5-2566-4b38-8cb3-9938b33c1401 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.379128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3792dbe5-2566-4b38-8cb3-9938b33c1401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.403989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3792dbe5-2566-4b38-8cb3-9938b33c1401 None None] MFA Rules not processed for user `87829678318948f3b81bd107076bdfa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:52.405198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3792dbe5-2566-4b38-8cb3-9938b33c1401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:52.417769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1139/2282] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:37:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:37:52.429585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81411cbd-93e9-4ee1-814c-9bff383f977c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:52.443189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81411cbd-93e9-4ee1-814c-9bff383f977c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:52.447015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c649d2c-3c91-4f63-8643-42fb73f05dcc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:52.447617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c649d2c-3c91-4f63-8643-42fb73f05dcc service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.447890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c649d2c-3c91-4f63-8643-42fb73f05dcc service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.448160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c649d2c-3c91-4f63-8643-42fb73f05dcc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.457349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c649d2c-3c91-4f63-8643-42fb73f05dcc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:52.458827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c649d2c-3c91-4f63-8643-42fb73f05dcc service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:52.467657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1144/2283] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:52.564094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ed086-b6af-42c4-bffb-2aa58ab2842b None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.564383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ed086-b6af-42c4-bffb-2aa58ab2842b None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.564663 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ed086-b6af-42c4-bffb-2aa58ab2842b None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.566606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1140/2284] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:37:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:37:52.572618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.572910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.573184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.595947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:52.597201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:52.606911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1145/2285] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:37:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:37:52.619562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ed086-b6af-42c4-bffb-2aa58ab2842b None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:52.632856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ed086-b6af-42c4-bffb-2aa58ab2842b None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:52.636619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803671d1-f102-4a63-a195-4780162f6721 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:52.637236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803671d1-f102-4a63-a195-4780162f6721 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.637512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803671d1-f102-4a63-a195-4780162f6721 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.637783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803671d1-f102-4a63-a195-4780162f6721 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.649893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803671d1-f102-4a63-a195-4780162f6721 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:52.651387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803671d1-f102-4a63-a195-4780162f6721 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:52.663737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1141/2286] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:52.756428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:52.757805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:52.777429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:37:52.833278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc38dc1-408e-48c9-9732-df821390ab5e None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:52.838386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:52.839023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:52.839305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:52.839577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:52.841343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:52.855741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:52.857354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:52.869208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1146/2287] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:53.410333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ea2e16-33b0-4cfb-9c2c-c8d395d17d09 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:53.410888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ea2e16-33b0-4cfb-9c2c-c8d395d17d09 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:53.411842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ea2e16-33b0-4cfb-9c2c-c8d395d17d09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:53.440200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-02ea2e16-33b0-4cfb-9c2c-c8d395d17d09 None None] MFA Rules not processed for user `ddfa371a39b84f42bba4621ae3a99794`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:53.441418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-02ea2e16-33b0-4cfb-9c2c-c8d395d17d09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:53.448468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1142/2288] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:37:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:53.709191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:53.718861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87d0781-9324-45be-9eb4-c7274dca7e3d service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:53.722467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:53.723088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:53.723370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:53.723646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:53.725356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:53.735221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:53.736650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:53.744805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1147/2289] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:37:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:37:56.029394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65477e-56d8-4789-9e5e-50411da61bb4 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:56.029884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65477e-56d8-4789-9e5e-50411da61bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:56.029973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65477e-56d8-4789-9e5e-50411da61bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:56.050091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-af65477e-56d8-4789-9e5e-50411da61bb4 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:56.051335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-af65477e-56d8-4789-9e5e-50411da61bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:56.058766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1143/2290] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:37:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:56.072625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:56.084146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8665dec-c287-413e-8568-333db5c265a0 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:56.089116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:56.089651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:56.089926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:56.090197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:56.098855 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:56.100549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:56.109397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1148/2291] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:37:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:37:56.190119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61103515-e80e-49df-9623-c0a74345341a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:56.190505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61103515-e80e-49df-9623-c0a74345341a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:56.190848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61103515-e80e-49df-9623-c0a74345341a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:56.213617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-61103515-e80e-49df-9623-c0a74345341a None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:37:56.215576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-61103515-e80e-49df-9623-c0a74345341a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:37:56.229558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1144/2292] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:37:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:37:56.254300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:37:56.281731 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a18a5-bb56-49a6-8f47-b88e363f584d service swift] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:37:56.295615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:37:56.296546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:37:56.297519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:37:56.298389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:37:56.310791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:37:56.312684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:37:56.334458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1149/2293] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:37:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:38:05.707182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61103515-e80e-49df-9623-c0a74345341a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:05.708666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-61103515-e80e-49df-9623-c0a74345341a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:05.724646 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61103515-e80e-49df-9623-c0a74345341a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:05.730136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:05.730719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:05.731025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:05.731300 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:05.732935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:05.796885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:05.798462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:05.807699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1145/2294] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:38:05 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:38:06.318240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=43 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:38:11.610807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:11.610807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:11.628958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5c4157-b1af-400b-a743-888ef8cf8d0e service swift] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:11.641718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:11.643414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:11.644192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:11.644474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:11.646502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:11.659973 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:11.661933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:11.674788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1150/2295] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:38:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:38:15.803538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=55 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:38:21.670484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:38:33.406980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:33.419369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4fc672-094c-498b-9dde-77f0bbd622e5 service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:33.425329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698c2163-099e-4c17-bcdf-e1ba4c612e95 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:33.425963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698c2163-099e-4c17-bcdf-e1ba4c612e95 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:33.426242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698c2163-099e-4c17-bcdf-e1ba4c612e95 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:33.426540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698c2163-099e-4c17-bcdf-e1ba4c612e95 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:33.428952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-698c2163-099e-4c17-bcdf-e1ba4c612e95 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:33.442366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698c2163-099e-4c17-bcdf-e1ba4c612e95 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:33.444037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698c2163-099e-4c17-bcdf-e1ba4c612e95 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:33.455237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1146/2296] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:38:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:38:38.232953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:38.245090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0f67c0-ea62-47d3-b430-0240e607c4a9 service nova] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:38.250987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:38.251732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:38.252095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:38.252433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:38.254224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:38.328215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:38.329803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:38.339762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1151/2297] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:38:38 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:38:39.082681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda1ed69-1dc7-4d53-8c48-929193edcd56 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:39.083889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda1ed69-1dc7-4d53-8c48-929193edcd56 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:39.083889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda1ed69-1dc7-4d53-8c48-929193edcd56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:39.103656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-fda1ed69-1dc7-4d53-8c48-929193edcd56 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:38:39.104864 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fda1ed69-1dc7-4d53-8c48-929193edcd56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:39.112462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1147/2298] 10.222.0.33 () {58 vars in 967 bytes} [Thu Oct 6 04:38:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:38:39.124914 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:39.138762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba45338-9fb7-4f69-9488-58397168f07a service nova] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:39.143864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:39.144465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:39.144743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:39.145010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:39.154370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:39.156039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:39.164106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1152/2299] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:38:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:38:44.423001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda1ed69-1dc7-4d53-8c48-929193edcd56 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:44.438057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda1ed69-1dc7-4d53-8c48-929193edcd56 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:44.443766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:44.444396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:44.444673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:44.444972 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:44.446670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:44.506149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:44.507775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:44.516675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1148/2300] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:38:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:38:48.498394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:48.509496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc9a2f1-e61e-431c-9d25-ebff5967cfb1 service swift] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:48.513511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02822f29-614c-413b-a8b2-6f42b6fc655e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:48.514332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02822f29-614c-413b-a8b2-6f42b6fc655e service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:48.514661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02822f29-614c-413b-a8b2-6f42b6fc655e service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:48.514878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02822f29-614c-413b-a8b2-6f42b6fc655e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:48.526847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02822f29-614c-413b-a8b2-6f42b6fc655e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:48.528337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02822f29-614c-413b-a8b2-6f42b6fc655e service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:48.539874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1153/2301] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:38:48 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:38:50.625794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:38:50.639325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437d47e2-1c45-40c5-b5a9-3ace843e42b9 service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:38:50.647519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f36579f-411c-486e-9d53-1266666debe1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:38:50.649175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f36579f-411c-486e-9d53-1266666debe1 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:50.649685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f36579f-411c-486e-9d53-1266666debe1 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:50.650172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f36579f-411c-486e-9d53-1266666debe1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:50.652204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0f36579f-411c-486e-9d53-1266666debe1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:50.666771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f36579f-411c-486e-9d53-1266666debe1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:38:50.668606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f36579f-411c-486e-9d53-1266666debe1 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:38:50.681699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1149/2302] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:38:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:38:58.169744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b52b7d-8fd7-4752-b7eb-1ea46603da51 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:38:58.170222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b52b7d-8fd7-4752-b7eb-1ea46603da51 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:38:58.170320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b52b7d-8fd7-4752-b7eb-1ea46603da51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:38:58.190244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b9b52b7d-8fd7-4752-b7eb-1ea46603da51 None None] MFA Rules not processed for user `a58cd041c02d46dcb8acf6adb11ee9de`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:38:58.191161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b9b52b7d-8fd7-4752-b7eb-1ea46603da51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:38:58.202415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1154/2303] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:38:58 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:39:00.675345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=43 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:39:04.770835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f67afc-afd9-429c-b14f-162b292588c7 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:39:04.771456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f67afc-afd9-429c-b14f-162b292588c7 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:39:04.771557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f67afc-afd9-429c-b14f-162b292588c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:39:04.810161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b7f67afc-afd9-429c-b14f-162b292588c7 None None] MFA Rules not processed for user `f68d73b5efc74b3380429cf72e506905`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:39:04.811752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b7f67afc-afd9-429c-b14f-162b292588c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:39:04.821963 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1150/2304] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:39:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:39:08.184348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=43 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:39:14.799900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:39:57.982763 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da0c1ba-fa68-4b11-98c2-5a6bae604633 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:39:57.983296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da0c1ba-fa68-4b11-98c2-5a6bae604633 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:39:57.983407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da0c1ba-fa68-4b11-98c2-5a6bae604633 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:39:58.002898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5da0c1ba-fa68-4b11-98c2-5a6bae604633 None None] MFA Rules not processed for user `d574235b791e485aa64130c599539945`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:39:58.004869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5da0c1ba-fa68-4b11-98c2-5a6bae604633 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:39:58.058723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1155/2305] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:39:57 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:08.052299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=25 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:25.315732 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcb2916-0955-4c9b-ae4c-b56bf0202276 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:25.316302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcb2916-0955-4c9b-ae4c-b56bf0202276 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:25.316302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcb2916-0955-4c9b-ae4c-b56bf0202276 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:25.344959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bfcb2916-0955-4c9b-ae4c-b56bf0202276 None None] MFA Rules not processed for user `56c86f73f98a4fb2af787534fd47e2ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:25.346306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bfcb2916-0955-4c9b-ae4c-b56bf0202276 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:25.371671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1151/2306] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:25 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:34.851596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da0c1ba-fa68-4b11-98c2-5a6bae604633 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:34.888092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da0c1ba-fa68-4b11-98c2-5a6bae604633 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:34.891740 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273158a4-d786-49be-9ccb-f7247d10b51c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:34.892328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273158a4-d786-49be-9ccb-f7247d10b51c service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:34.892601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273158a4-d786-49be-9ccb-f7247d10b51c service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:34.892869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273158a4-d786-49be-9ccb-f7247d10b51c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:34.894615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-273158a4-d786-49be-9ccb-f7247d10b51c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:34.983127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273158a4-d786-49be-9ccb-f7247d10b51c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:34.984550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273158a4-d786-49be-9ccb-f7247d10b51c service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:35.004106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1156/2307] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:35.369868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=15 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:38.408389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de978b2b-3f2b-4a5c-83fc-25b1bcd30477 None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:38.408949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de978b2b-3f2b-4a5c-83fc-25b1bcd30477 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:38.408949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de978b2b-3f2b-4a5c-83fc-25b1bcd30477 None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:38.410391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1152/2308] 10.222.0.33 () {54 vars in 961 bytes} [Thu Oct 6 04:40:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:40:38.415923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a8d94d-d76c-432a-b85f-c3e7e2c7d24c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:38.416398 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a8d94d-d76c-432a-b85f-c3e7e2c7d24c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:38.416718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a8d94d-d76c-432a-b85f-c3e7e2c7d24c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:38.483142 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de978b2b-3f2b-4a5c-83fc-25b1bcd30477 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:38.486333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-de978b2b-3f2b-4a5c-83fc-25b1bcd30477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:38.505970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de978b2b-3f2b-4a5c-83fc-25b1bcd30477 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:38.506844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-05a8d94d-d76c-432a-b85f-c3e7e2c7d24c None None] MFA Rules not processed for user `7e2fe69b3b3b4ba5b55a4698b3ed70b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:38.508165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-05a8d94d-d76c-432a-b85f-c3e7e2c7d24c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:38.511542 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:38.512192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:38.512451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:38.512719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] PATH_INFO: `/v3/users/a58cd041c02d46dcb8acf6adb11ee9de` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:38.513828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a58cd041c02d46dcb8acf6adb11ee9de)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:38.515990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:38.529953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1157/2309] 10.222.0.33 () {58 vars in 1058 bytes} [Thu Oct 6 04:40:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:38.539484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a8d94d-d76c-432a-b85f-c3e7e2c7d24c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:38.549817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a8d94d-d76c-432a-b85f-c3e7e2c7d24c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:38.557412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:38.558053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:38.558325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:38.558758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:38.560747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:38.577535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:38.579182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:38.587957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a58cd041c02d46dcb8acf6adb11ee9de'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.588640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a58cd041c02d46dcb8acf6adb11ee9de'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.597309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a58cd041c02d46dcb8acf6adb11ee9de'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.600905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1158/2310] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:40:38.615431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a58cd041c02d46dcb8acf6adb11ee9de'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.616571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-23649353-79fa-4c73-ad4a-3886b779a495 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a58cd041c02d46dcb8acf6adb11ee9de'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.638153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1153/2311] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:38 2022] DELETE /identity/v3/users/a58cd041c02d46dcb8acf6adb11ee9de => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:40:38.823083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:38.833956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e9b377-598c-4053-a0dc-a132def25775 service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:38.853529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:38.892743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:38.893362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:38.893641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:38.893913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] PATH_INFO: `/v3/projects/0d643fb7b9314f0a9bb8f2c37f06bb2d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:38.895042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d643fb7b9314f0a9bb8f2c37f06bb2d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:38.897023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:38.940729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d643fb7b9314f0a9bb8f2c37f06bb2d', 'request_id': 'req-53887ec0-0b77-4851-8f60-babb9afdce3e'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.946754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d643fb7b9314f0a9bb8f2c37f06bb2d', 'request_id': 'req-53887ec0-0b77-4851-8f60-babb9afdce3e'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.952798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d643fb7b9314f0a9bb8f2c37f06bb2d', 'request_id': 'req-53887ec0-0b77-4851-8f60-babb9afdce3e'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.964422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] The token cache is being invalidate because project 0d643fb7b9314f0a9bb8f2c37f06bb2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:38.964706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.965361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-53887ec0-0b77-4851-8f60-babb9afdce3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:38.966829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1159/2312] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:38 2022] DELETE /identity/v3/projects/0d643fb7b9314f0a9bb8f2c37f06bb2d => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:38.971703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7d3bcf-ed25-4986-a841-26b8ba5b7ca0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:38.971990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7d3bcf-ed25-4986-a841-26b8ba5b7ca0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:38.972263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7d3bcf-ed25-4986-a841-26b8ba5b7ca0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.026469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3a7d3bcf-ed25-4986-a841-26b8ba5b7ca0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:39.028001 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3a7d3bcf-ed25-4986-a841-26b8ba5b7ca0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:39.104056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1154/2313] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:39.107020 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8714b1-dd09-42d9-b220-ab58f5c90150 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.107637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8714b1-dd09-42d9-b220-ab58f5c90150 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.108030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8714b1-dd09-42d9-b220-ab58f5c90150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.140974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5f8714b1-dd09-42d9-b220-ab58f5c90150 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:39.142391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5f8714b1-dd09-42d9-b220-ab58f5c90150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:39.151031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1160/2314] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:39.155427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7d3bcf-ed25-4986-a841-26b8ba5b7ca0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.166684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7d3bcf-ed25-4986-a841-26b8ba5b7ca0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.171446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.172042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.172822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.172822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.173246 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:40:39.173547 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.175161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.188480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1155/2315] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:40:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:40:39.192536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8714b1-dd09-42d9-b220-ab58f5c90150 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.202543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8714b1-dd09-42d9-b220-ab58f5c90150 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.207273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.207868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.208151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.208572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.209274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.216321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.247519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1161/2316] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:39 2022] POST /identity/v3/projects => generated 371 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:39.252064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.265028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce28917-fc80-4872-9d6b-03ec2c88780a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.298226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:39.325197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.325794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.326072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.326343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.326989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.328898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.350273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1156/2317] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:39 2022] POST /identity/v3/users => generated 286 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:39.354532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.365481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0425ceb-1847-471d-9095-efc7401589d2 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.370511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.371136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.371414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.371683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.372324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.373922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.384579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1162/2318] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:39.388477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.398345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33786e7a-7fb7-4b78-a63c-f880cba34f00 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.403009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2e8a6f-634b-49dc-96c7-7dbbac6a2336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.403596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2e8a6f-634b-49dc-96c7-7dbbac6a2336 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.403900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2e8a6f-634b-49dc-96c7-7dbbac6a2336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.404238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2e8a6f-634b-49dc-96c7-7dbbac6a2336 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.404856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2e8a6f-634b-49dc-96c7-7dbbac6a2336 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.406427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2e8a6f-634b-49dc-96c7-7dbbac6a2336 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.416730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1157/2319] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:39.420785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.430805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff71439a-e7b3-4354-aa19-6e1339294b91 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.435488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.436119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.436392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.436658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] PATH_INFO: `/v3/projects/5b5f142e2dff4ea2b04101672b9e5dc7/users/52965ae62a5944ff9af68c6a37a26d4d/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.450688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b5f142e2dff4ea2b04101672b9e5dc7, user_id=52965ae62a5944ff9af68c6a37a26d4d, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.453681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.467249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1163/2320] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:39 2022] PUT /identity/v3/projects/5b5f142e2dff4ea2b04101672b9e5dc7/users/52965ae62a5944ff9af68c6a37a26d4d/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:39.478399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33614b0f-9f56-4c72-9c2e-43be43644f2a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.478686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33614b0f-9f56-4c72-9c2e-43be43644f2a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.478989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33614b0f-9f56-4c72-9c2e-43be43644f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.506415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-33614b0f-9f56-4c72-9c2e-43be43644f2a None None] MFA Rules not processed for user `52965ae62a5944ff9af68c6a37a26d4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:39.507723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-33614b0f-9f56-4c72-9c2e-43be43644f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:39.572886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1158/2321] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:39 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:39.577347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.625978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:39.640981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03987d9-75ad-4b7f-b372-432bbb456ef1 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.645688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.646876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.647183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.647452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.648097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.649996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.677042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1164/2322] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:39 2022] POST /identity/v3/projects => generated 371 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:39.681675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33614b0f-9f56-4c72-9c2e-43be43644f2a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.693837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33614b0f-9f56-4c72-9c2e-43be43644f2a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.752807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.753493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.753740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.754010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.754680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.757394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.780304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1159/2323] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:39 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:39.784165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.794716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc64fa1-8a8d-46d3-a227-25d8749a138e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.800150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.801145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.806802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.807180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.807854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.809578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.825577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1165/2324] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:39 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:39.826784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.837726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de92ac6b-9a2b-490b-8044-2c1df2bcf2f0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.842867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.843555 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.843968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.844251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] PATH_INFO: `/v3/projects/f23e5280e6fa4f39bccfa0563586898e/users/bc6efcd4e5554cfb94597005e2b9e025/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.862778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f23e5280e6fa4f39bccfa0563586898e, user_id=bc6efcd4e5554cfb94597005e2b9e025, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.866352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.880404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1160/2325] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:39 2022] PUT /identity/v3/projects/f23e5280e6fa4f39bccfa0563586898e/users/bc6efcd4e5554cfb94597005e2b9e025/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:39.884735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.894693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c497c938-4794-4083-8573-d03544fcd3cd admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.954640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7945c746-851a-4b89-8a9d-4e8b1cb07a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.955296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7945c746-851a-4b89-8a9d-4e8b1cb07a3f admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.955578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7945c746-851a-4b89-8a9d-4e8b1cb07a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.955850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7945c746-851a-4b89-8a9d-4e8b1cb07a3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.956525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7945c746-851a-4b89-8a9d-4e8b1cb07a3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.958113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7945c746-851a-4b89-8a9d-4e8b1cb07a3f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:39.969032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1166/2326] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:39 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:39.973335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:39.984290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91423e34-593d-4d9c-b2a4-7ea56195f696 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:39.988978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:39.989687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:39.990737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:39.991248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] PATH_INFO: `/v3/projects/f23e5280e6fa4f39bccfa0563586898e/users/bc6efcd4e5554cfb94597005e2b9e025/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:39.993562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f23e5280e6fa4f39bccfa0563586898e, user_id=bc6efcd4e5554cfb94597005e2b9e025, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:39.996505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:40.002584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:40.008278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1161/2327] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:39 2022] PUT /identity/v3/projects/f23e5280e6fa4f39bccfa0563586898e/users/bc6efcd4e5554cfb94597005e2b9e025/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:40.018827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134a5e33-0029-4a37-b6c9-3465fff5b423 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:40.019142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134a5e33-0029-4a37-b6c9-3465fff5b423 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:40.019417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134a5e33-0029-4a37-b6c9-3465fff5b423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:40.047566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-134a5e33-0029-4a37-b6c9-3465fff5b423 None None] MFA Rules not processed for user `bc6efcd4e5554cfb94597005e2b9e025`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:40.048780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-134a5e33-0029-4a37-b6c9-3465fff5b423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:40.116116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1167/2328] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:40 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:40.129148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:40.130998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:40.193609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c9edaf-2bf9-419d-996e-6d1bd88b00e2 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:40.197306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4868f8b8-dc52-4841-a7ff-59b870bd252c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:40.197911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4868f8b8-dc52-4841-a7ff-59b870bd252c service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:40.198314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4868f8b8-dc52-4841-a7ff-59b870bd252c service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:40.199082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4868f8b8-dc52-4841-a7ff-59b870bd252c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:40.214103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4868f8b8-dc52-4841-a7ff-59b870bd252c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:40.214103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4868f8b8-dc52-4841-a7ff-59b870bd252c service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:40.223209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1162/2329] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:40 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:40.630841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134a5e33-0029-4a37-b6c9-3465fff5b423 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:40.676930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:40.696166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134a5e33-0029-4a37-b6c9-3465fff5b423 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:40.700715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:40.701373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:40.701649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:40.701921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] PATH_INFO: `/v3/users/52965ae62a5944ff9af68c6a37a26d4d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:40.703071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=52965ae62a5944ff9af68c6a37a26d4d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:40.705008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:40.760835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52965ae62a5944ff9af68c6a37a26d4d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:40.762038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52965ae62a5944ff9af68c6a37a26d4d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:40.770527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52965ae62a5944ff9af68c6a37a26d4d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:40.771450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52965ae62a5944ff9af68c6a37a26d4d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:40.784642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52965ae62a5944ff9af68c6a37a26d4d'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:40.804146 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1168/2330] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:40 2022] DELETE /identity/v3/users/52965ae62a5944ff9af68c6a37a26d4d => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:40.805957 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:40.806453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:40.806571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:40.827122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:40.828373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:40.888861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1163/2331] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:40.898951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:40.900821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:40.976038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e8801a-d45c-42eb-b5d9-5e7e8190f2cb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:40.979703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:40.980307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:40.980581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:40.980850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:40.990295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:40.991976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:41.011047 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1169/2332] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:41.449855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:41.451604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:41.464587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb145e0-7c56-4541-90c9-848c912f4614 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:41.481068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:41.481849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:41.482262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:41.482629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] PATH_INFO: `/v3/projects/5b5f142e2dff4ea2b04101672b9e5dc7` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:41.484103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b5f142e2dff4ea2b04101672b9e5dc7)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:41.486108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:41.538543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b5f142e2dff4ea2b04101672b9e5dc7', 'request_id': 'req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.545919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b5f142e2dff4ea2b04101672b9e5dc7', 'request_id': 'req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.551869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b5f142e2dff4ea2b04101672b9e5dc7', 'request_id': 'req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.565333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] The token cache is being invalidate because project 5b5f142e2dff4ea2b04101672b9e5dc7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:41.565601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.566199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.568266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1164/2333] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:41 2022] DELETE /identity/v3/projects/5b5f142e2dff4ea2b04101672b9e5dc7 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:41.572878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:41.596488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7c02f9-9de9-4af9-a086-5f2e62d7eb8c service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:41.658657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:41.666626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:41.667291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:41.667814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:41.668316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] PATH_INFO: `/v3/users/bc6efcd4e5554cfb94597005e2b9e025` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:41.669635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc6efcd4e5554cfb94597005e2b9e025)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:41.672375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:41.727908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc6efcd4e5554cfb94597005e2b9e025'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.728651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc6efcd4e5554cfb94597005e2b9e025'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.736105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc6efcd4e5554cfb94597005e2b9e025'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.736732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc6efcd4e5554cfb94597005e2b9e025'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.758490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3cda4c8c-30ee-48da-9111-ab54af6439c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc6efcd4e5554cfb94597005e2b9e025'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:41.778153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1170/2334] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:41 2022] DELETE /identity/v3/users/bc6efcd4e5554cfb94597005e2b9e025 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:42.101030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.102528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:42.120179 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3b426e-fc2d-43af-ba9f-d9f4b692bece admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.144577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:42.196991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.197621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.197916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.198223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] PATH_INFO: `/v3/projects/f23e5280e6fa4f39bccfa0563586898e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.199421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f23e5280e6fa4f39bccfa0563586898e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.201740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.272952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f23e5280e6fa4f39bccfa0563586898e', 'request_id': 'req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:42.279778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f23e5280e6fa4f39bccfa0563586898e', 'request_id': 'req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:42.288813 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f23e5280e6fa4f39bccfa0563586898e', 'request_id': 'req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:42.304220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] The token cache is being invalidate because project f23e5280e6fa4f39bccfa0563586898e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:42.304220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:42.304493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-2c0f73bd-e6ab-4313-8d22-0f35fbb1abb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:42.305915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1165/2335] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:42 2022] DELETE /identity/v3/projects/f23e5280e6fa4f39bccfa0563586898e => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:42.310804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557a4d1e-f5db-4531-8700-4d2257aa7d8f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.311211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557a4d1e-f5db-4531-8700-4d2257aa7d8f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.311489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557a4d1e-f5db-4531-8700-4d2257aa7d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.331706 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-557a4d1e-f5db-4531-8700-4d2257aa7d8f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:42.332921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-557a4d1e-f5db-4531-8700-4d2257aa7d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:42.393314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1171/2336] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:40:42.402040 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02ff3df-a0b5-476e-a765-a53fae965d94 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.402345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02ff3df-a0b5-476e-a765-a53fae965d94 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.402622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02ff3df-a0b5-476e-a765-a53fae965d94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.424233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-e02ff3df-a0b5-476e-a765-a53fae965d94 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:42.425469 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e02ff3df-a0b5-476e-a765-a53fae965d94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:42.434834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1166/2337] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:42.439141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557a4d1e-f5db-4531-8700-4d2257aa7d8f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.450127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557a4d1e-f5db-4531-8700-4d2257aa7d8f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.454812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.455447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.455724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.455990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.456635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:40:42.456936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.458671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.470881 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1172/2338] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:40:42.475511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02ff3df-a0b5-476e-a765-a53fae965d94 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.484902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02ff3df-a0b5-476e-a765-a53fae965d94 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.490321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.491038 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.491317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.491606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.492266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.494466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.523840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1167/2339] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:42 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:42.528305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.539420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826878fb-ecef-4790-bb05-d233b05f610e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.601275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.601885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.602165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.602433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.603137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.605312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.614320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:42.633143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1173/2340] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:42 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:42.634382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.644853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fbf78-e1d8-4871-bb0f-eccff29004d7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.649876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.650467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.650740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.651026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.651664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.653237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.664288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1168/2341] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:40:42.668122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.678716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b359a1b-3b3d-4d9b-af3d-d80ba80b7286 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.683464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1e752b-4ee1-4548-8888-69f83a42ae83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.684065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1e752b-4ee1-4548-8888-69f83a42ae83 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.684339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1e752b-4ee1-4548-8888-69f83a42ae83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.684607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1e752b-4ee1-4548-8888-69f83a42ae83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.685247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1e752b-4ee1-4548-8888-69f83a42ae83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.686838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1e752b-4ee1-4548-8888-69f83a42ae83 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.697859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1174/2342] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:42.702096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.713379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89236b05-8f1d-4ca5-ace0-e632a9bc4eb2 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.718014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.718660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.718933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.719238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] PATH_INFO: `/v3/projects/77933eb4333e4b3d9854f0b91ef624b2/users/9efdf07cb46f4fe78b89985efb7fc906/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.735400 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77933eb4333e4b3d9854f0b91ef624b2, user_id=9efdf07cb46f4fe78b89985efb7fc906, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.738198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.753466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1169/2343] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:42 2022] PUT /identity/v3/projects/77933eb4333e4b3d9854f0b91ef624b2/users/9efdf07cb46f4fe78b89985efb7fc906/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:42.764579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b007dca-6b80-4b54-b283-dece77756649 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.764848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b007dca-6b80-4b54-b283-dece77756649 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.765159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b007dca-6b80-4b54-b283-dece77756649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.794103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1b007dca-6b80-4b54-b283-dece77756649 None None] MFA Rules not processed for user `9efdf07cb46f4fe78b89985efb7fc906`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:42.795348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1b007dca-6b80-4b54-b283-dece77756649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:42.859758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1175/2344] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:42.864201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.915079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:42.927122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ed3e77-fdbb-4311-b884-53153419ddf7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:42.932124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:42.932713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:42.932996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:42.933266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:42.933895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:42.935812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:42.961886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1170/2345] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:42 2022] POST /identity/v3/projects => generated 385 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:42.965908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b007dca-6b80-4b54-b283-dece77756649 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:42.977996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b007dca-6b80-4b54-b283-dece77756649 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:43.036063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:43.036594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:43.036872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:43.037143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:43.038041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:43.040175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:43.064499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1176/2346] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:42 2022] POST /identity/v3/users => generated 299 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:43.065884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:43.076989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269cf4f6-8017-46a2-bce6-ec6a0f1ac4a5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:43.081783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:43.082373 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:43.082650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:43.082920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:43.084260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:43.085830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:43.096675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1171/2347] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:43.100912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:43.111102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965025ba-66bd-4e0d-b7b3-4ab44d266efd admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:43.115809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:43.116449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:43.116727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:43.116998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] PATH_INFO: `/v3/projects/dbcb041b9979400bab5ec8d342bc743b/users/0404c16c01b042b6b3451861d00a66bc/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:43.132774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbcb041b9979400bab5ec8d342bc743b, user_id=0404c16c01b042b6b3451861d00a66bc, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:43.135539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:43.148831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1177/2348] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:43 2022] PUT /identity/v3/projects/dbcb041b9979400bab5ec8d342bc743b/users/0404c16c01b042b6b3451861d00a66bc/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:43.152791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:43.162698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5cddd3-5fc4-466e-af04-c0a72523b13c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:43.217674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14c5eba-fbad-4d23-9bb3-9fd008c38b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:43.218306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14c5eba-fbad-4d23-9bb3-9fd008c38b8d admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:43.218606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14c5eba-fbad-4d23-9bb3-9fd008c38b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:43.218908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14c5eba-fbad-4d23-9bb3-9fd008c38b8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:43.219839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14c5eba-fbad-4d23-9bb3-9fd008c38b8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:43.221422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14c5eba-fbad-4d23-9bb3-9fd008c38b8d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:43.231742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1172/2349] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:43.235943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:43.246865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b5ec6d-d364-4c67-b8cd-fe7ce48246cd admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:43.251369 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:43.252006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:43.252277 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:43.252545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] PATH_INFO: `/v3/projects/dbcb041b9979400bab5ec8d342bc743b/users/0404c16c01b042b6b3451861d00a66bc/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:43.254391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbcb041b9979400bab5ec8d342bc743b, user_id=0404c16c01b042b6b3451861d00a66bc, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:43.257803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:43.267650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1178/2350] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:43 2022] PUT /identity/v3/projects/dbcb041b9979400bab5ec8d342bc743b/users/0404c16c01b042b6b3451861d00a66bc/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:43.279104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce631f9-cfb4-4fbd-a86c-e5a1a904ee86 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:43.279521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce631f9-cfb4-4fbd-a86c-e5a1a904ee86 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:43.279799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce631f9-cfb4-4fbd-a86c-e5a1a904ee86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:43.307890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7ce631f9-cfb4-4fbd-a86c-e5a1a904ee86 None None] MFA Rules not processed for user `0404c16c01b042b6b3451861d00a66bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:43.309042 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7ce631f9-cfb4-4fbd-a86c-e5a1a904ee86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:43.375344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1173/2351] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:43 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:43.386048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:43.387759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:43.413519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:43.467183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ed36c3-473d-4de7-962f-890e3f8a62ca admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:43.470897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6e5988-6192-4c51-bc6a-9cb9f39b5503 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:43.471513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6e5988-6192-4c51-bc6a-9cb9f39b5503 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:43.471786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6e5988-6192-4c51-bc6a-9cb9f39b5503 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:43.472054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6e5988-6192-4c51-bc6a-9cb9f39b5503 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:43.480903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6e5988-6192-4c51-bc6a-9cb9f39b5503 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:43.482320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6e5988-6192-4c51-bc6a-9cb9f39b5503 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:43.492060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1179/2352] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:43 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:44.028119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce631f9-cfb4-4fbd-a86c-e5a1a904ee86 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:44.075968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:44.091849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce631f9-cfb4-4fbd-a86c-e5a1a904ee86 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:44.096650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:44.097121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:44.097398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:44.097669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] PATH_INFO: `/v3/users/9efdf07cb46f4fe78b89985efb7fc906` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:44.098828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9efdf07cb46f4fe78b89985efb7fc906)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:44.100964 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:44.150120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9efdf07cb46f4fe78b89985efb7fc906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.150767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9efdf07cb46f4fe78b89985efb7fc906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.157689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9efdf07cb46f4fe78b89985efb7fc906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.169348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9efdf07cb46f4fe78b89985efb7fc906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.169980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9efdf07cb46f4fe78b89985efb7fc906'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.182761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1174/2353] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:44 2022] DELETE /identity/v3/users/9efdf07cb46f4fe78b89985efb7fc906 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:44.187310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:44.187757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:44.187985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:44.207197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:44.208506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:44.267728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1180/2354] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:44.277465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:44.278906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:44.335178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9ae97-bee4-45b4-8fbf-993f98a1e0b1 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:44.338807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:44.339446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:44.339716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:44.339986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:44.349558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:44.350939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:44.360503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1175/2355] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:44.618434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:44.620497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:44.633666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e887763-a60b-4445-a282-9198f1cd1159 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:44.638409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:44.639080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:44.639357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:44.639627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] PATH_INFO: `/v3/projects/77933eb4333e4b3d9854f0b91ef624b2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:44.640757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77933eb4333e4b3d9854f0b91ef624b2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:44.642676 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:44.689778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77933eb4333e4b3d9854f0b91ef624b2', 'request_id': 'req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.695992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77933eb4333e4b3d9854f0b91ef624b2', 'request_id': 'req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.702078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77933eb4333e4b3d9854f0b91ef624b2', 'request_id': 'req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.715543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] The token cache is being invalidate because project 77933eb4333e4b3d9854f0b91ef624b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:44.715810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.716427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.717898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1181/2356] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:44 2022] DELETE /identity/v3/projects/77933eb4333e4b3d9854f0b91ef624b2 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:44.721849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:44.732295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1038b113-8b20-4d81-ba87-d562fb4b844e service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:44.791752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:44.792363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:44.792639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:44.792908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] PATH_INFO: `/v3/users/0404c16c01b042b6b3451861d00a66bc` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:44.794010 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0404c16c01b042b6b3451861d00a66bc)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:44.795991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:44.805861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:44.842274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0404c16c01b042b6b3451861d00a66bc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.842902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0404c16c01b042b6b3451861d00a66bc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.850416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0404c16c01b042b6b3451861d00a66bc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.862159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0404c16c01b042b6b3451861d00a66bc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.862730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-5e5f5a94-fc36-4d41-a219-3f6491923d50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0404c16c01b042b6b3451861d00a66bc'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:44.875338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1176/2357] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:44 2022] DELETE /identity/v3/users/0404c16c01b042b6b3451861d00a66bc => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:45.141078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.142561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:45.161648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b7421c-eca9-4a5d-9f8a-2afb6d2d26e4 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.193583 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:45.232959 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.233645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.233921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.234191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] PATH_INFO: `/v3/projects/dbcb041b9979400bab5ec8d342bc743b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.235326 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbcb041b9979400bab5ec8d342bc743b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.237272 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.284106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbcb041b9979400bab5ec8d342bc743b', 'request_id': 'req-6a550dad-8007-4416-9f47-1a7eb7add36c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:45.290071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbcb041b9979400bab5ec8d342bc743b', 'request_id': 'req-6a550dad-8007-4416-9f47-1a7eb7add36c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:45.296261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbcb041b9979400bab5ec8d342bc743b', 'request_id': 'req-6a550dad-8007-4416-9f47-1a7eb7add36c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:45.308982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] The token cache is being invalidate because project dbcb041b9979400bab5ec8d342bc743b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:45.309263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:45.309958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6a550dad-8007-4416-9f47-1a7eb7add36c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:45.311443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1182/2358] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:45 2022] DELETE /identity/v3/projects/dbcb041b9979400bab5ec8d342bc743b => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:45.317226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec065c-1867-489b-a23c-0ce4ac2a5238 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.318287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec065c-1867-489b-a23c-0ce4ac2a5238 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.318287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec065c-1867-489b-a23c-0ce4ac2a5238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.337754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-56ec065c-1867-489b-a23c-0ce4ac2a5238 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:45.338926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-56ec065c-1867-489b-a23c-0ce4ac2a5238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:45.396716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1177/2359] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:45.404447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219171ee-8e6d-46f4-af34-e272c78e2915 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.404728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219171ee-8e6d-46f4-af34-e272c78e2915 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.405025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219171ee-8e6d-46f4-af34-e272c78e2915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.424586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-219171ee-8e6d-46f4-af34-e272c78e2915 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:45.425779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-219171ee-8e6d-46f4-af34-e272c78e2915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:45.434080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1183/2360] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:45.438308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec065c-1867-489b-a23c-0ce4ac2a5238 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.448520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec065c-1867-489b-a23c-0ce4ac2a5238 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.453100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.453687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.453965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.454236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.454871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:40:45.455199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.456783 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.468471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1178/2361] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:40:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:40:45.472653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219171ee-8e6d-46f4-af34-e272c78e2915 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.481931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219171ee-8e6d-46f4-af34-e272c78e2915 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.486476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.487098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.487374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.487641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.488271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.490643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.518371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1184/2362] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:45 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:45.523228 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.533102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb6f0f7-390a-4adc-8bf3-b2f05b308a52 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.588327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.588910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.589185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.589455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.590079 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.592007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.612690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1179/2363] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:45 2022] POST /identity/v3/users => generated 287 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:45.617646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.628199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4be3523-6c91-4caa-a43c-ba7f148ac99c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.632797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.633380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.633653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.633920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.634551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.636164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.647324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1185/2364] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:45.651080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.657391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:45.661245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89cd4e8-d379-46b9-8dcf-7bc662e1480b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.667831 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be548f08-6513-4c0c-90a9-87de7ac05bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.668619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be548f08-6513-4c0c-90a9-87de7ac05bfc admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.669058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be548f08-6513-4c0c-90a9-87de7ac05bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.669383 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be548f08-6513-4c0c-90a9-87de7ac05bfc admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.670155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be548f08-6513-4c0c-90a9-87de7ac05bfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.671767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be548f08-6513-4c0c-90a9-87de7ac05bfc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.682405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1180/2365] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:45.686465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.696487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec232e-7b29-4ce2-9ce8-c215ec1013d9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.701178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.701807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.702079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.702348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] PATH_INFO: `/v3/projects/81772441e71a45758a6f45cb642e867c/users/1b3ce18d844644d19e4aad0540f3ab9e/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.717153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] RBAC: Authorizing `identity:create_grant(project_id=81772441e71a45758a6f45cb642e867c, user_id=1b3ce18d844644d19e4aad0540f3ab9e, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.719823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.734399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1186/2366] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:45 2022] PUT /identity/v3/projects/81772441e71a45758a6f45cb642e867c/users/1b3ce18d844644d19e4aad0540f3ab9e/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:45.745916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45599158-44d9-46d2-8950-8667f0dc2baf None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.746201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45599158-44d9-46d2-8950-8667f0dc2baf None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.746481 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45599158-44d9-46d2-8950-8667f0dc2baf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.775852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-45599158-44d9-46d2-8950-8667f0dc2baf None None] MFA Rules not processed for user `1b3ce18d844644d19e4aad0540f3ab9e`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:45.777078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-45599158-44d9-46d2-8950-8667f0dc2baf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:45.842138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1181/2367] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:45.853457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:45.855178 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:45.911863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59369ee7-1a28-49df-a507-f554d9de364d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:45.915403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3037c86-ed35-4cef-94f1-44d4ed5dc683 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:45.915995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3037c86-ed35-4cef-94f1-44d4ed5dc683 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:45.916299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3037c86-ed35-4cef-94f1-44d4ed5dc683 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:45.916569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3037c86-ed35-4cef-94f1-44d4ed5dc683 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:45.924654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3037c86-ed35-4cef-94f1-44d4ed5dc683 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:45.926068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3037c86-ed35-4cef-94f1-44d4ed5dc683 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:45.930314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:45.937304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1187/2368] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:48.977244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45599158-44d9-46d2-8950-8667f0dc2baf None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:49.039117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45599158-44d9-46d2-8950-8667f0dc2baf None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:49.043640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:49.044252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.044529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.044800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] PATH_INFO: `/v3/users/1b3ce18d844644d19e4aad0540f3ab9e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.045905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b3ce18d844644d19e4aad0540f3ab9e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:49.047845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:49.092907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b3ce18d844644d19e4aad0540f3ab9e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.093558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b3ce18d844644d19e4aad0540f3ab9e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.101063 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b3ce18d844644d19e4aad0540f3ab9e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.112837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b3ce18d844644d19e4aad0540f3ab9e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.113460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b3ce18d844644d19e4aad0540f3ab9e'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.125953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1182/2369] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:48 2022] DELETE /identity/v3/users/1b3ce18d844644d19e4aad0540f3ab9e => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:49.132287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.132801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.132904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.151608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:49.152786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:49.213872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1188/2370] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:49.224523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:49.225937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:49.231479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:49.283982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cce0080-1c8e-4103-9026-dcdb5c0ac7d5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:49.288309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eada0d0-b880-4710-a549-51fb892818ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:49.288901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eada0d0-b880-4710-a549-51fb892818ca service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.289175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eada0d0-b880-4710-a549-51fb892818ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.289444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eada0d0-b880-4710-a549-51fb892818ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.299225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eada0d0-b880-4710-a549-51fb892818ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:49.300717 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eada0d0-b880-4710-a549-51fb892818ca service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:49.310706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1183/2371] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:49.587690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:49.589143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:49.602525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436fcda9-9ea7-4343-9627-5edb3a3aae36 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:49.607440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:49.608094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.608372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.608644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] PATH_INFO: `/v3/projects/81772441e71a45758a6f45cb642e867c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.609776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81772441e71a45758a6f45cb642e867c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:49.611809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:49.663194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81772441e71a45758a6f45cb642e867c', 'request_id': 'req-42471dd8-e435-4fd7-bf62-ed982afb6f21'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.668678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81772441e71a45758a6f45cb642e867c', 'request_id': 'req-42471dd8-e435-4fd7-bf62-ed982afb6f21'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.675006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81772441e71a45758a6f45cb642e867c', 'request_id': 'req-42471dd8-e435-4fd7-bf62-ed982afb6f21'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.688255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] The token cache is being invalidate because project 81772441e71a45758a6f45cb642e867c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:49.688547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.689164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-42471dd8-e435-4fd7-bf62-ed982afb6f21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:49.690998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1189/2372] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:49 2022] DELETE /identity/v3/projects/81772441e71a45758a6f45cb642e867c => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:49.696031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b666a20-6005-4b94-9d5b-d11f9dcfea5d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.696312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b666a20-6005-4b94-9d5b-d11f9dcfea5d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.696583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b666a20-6005-4b94-9d5b-d11f9dcfea5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.716636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-9b666a20-6005-4b94-9d5b-d11f9dcfea5d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:49.717824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9b666a20-6005-4b94-9d5b-d11f9dcfea5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:49.778115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1184/2373] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:49.786876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fc3fc-8e69-4fa5-8d93-99e473c82342 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.787354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fc3fc-8e69-4fa5-8d93-99e473c82342 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.787698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fc3fc-8e69-4fa5-8d93-99e473c82342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.807958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-cd2fc3fc-8e69-4fa5-8d93-99e473c82342 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:49.809174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cd2fc3fc-8e69-4fa5-8d93-99e473c82342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:49.817921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1190/2374] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:49.822198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b666a20-6005-4b94-9d5b-d11f9dcfea5d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:49.831860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b666a20-6005-4b94-9d5b-d11f9dcfea5d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:49.837117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:49.837703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.837982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.838289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.838940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:40:49.839266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:49.840853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:49.852189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1185/2375] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:40:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:40:49.856259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2fc3fc-8e69-4fa5-8d93-99e473c82342 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:49.865894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2fc3fc-8e69-4fa5-8d93-99e473c82342 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:49.870916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:49.871778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.871883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.872064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.872696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:49.874610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:49.906106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1191/2376] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:49 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:49.910119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:49.920713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf49dba-39b7-4ee1-a8cb-26c6bd86d6f8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:49.981012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:49.981608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:49.981880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:49.982148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:49.982773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:49.984704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.010606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1186/2377] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:49 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:40:50.014984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.025975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844626f8-99ec-46f5-bed9-6d540f190e17 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.030622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.031170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.031448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.031717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.032358 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.033929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.044772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1192/2378] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:50.048512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.059190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16717cd1-5876-4d05-b93a-3b779761daf0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.064239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78abce46-50de-4126-9b7c-e99c7a1a20b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.064865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abce46-50de-4126-9b7c-e99c7a1a20b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.065155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abce46-50de-4126-9b7c-e99c7a1a20b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.065417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78abce46-50de-4126-9b7c-e99c7a1a20b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.066109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abce46-50de-4126-9b7c-e99c7a1a20b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.067997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78abce46-50de-4126-9b7c-e99c7a1a20b9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.079943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1187/2379] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:40:50.084851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.094794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11938b39-192c-498f-b18d-5bb7bd31534f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.101926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.102882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.103494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.103950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] PATH_INFO: `/v3/projects/42f57370bf3a4d3fb7499005ff555cba/users/3c66003363624d3bb09a7b9be6e6193f/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.121791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=42f57370bf3a4d3fb7499005ff555cba, user_id=3c66003363624d3bb09a7b9be6e6193f, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.124494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.139275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1193/2380] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:50 2022] PUT /identity/v3/projects/42f57370bf3a4d3fb7499005ff555cba/users/3c66003363624d3bb09a7b9be6e6193f/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:40:50.152024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ee1e4d-5a96-45c2-a188-20f3dd840227 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.152521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ee1e4d-5a96-45c2-a188-20f3dd840227 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.152923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ee1e4d-5a96-45c2-a188-20f3dd840227 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.174292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:50.182989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-96ee1e4d-5a96-45c2-a188-20f3dd840227 None None] MFA Rules not processed for user `3c66003363624d3bb09a7b9be6e6193f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:50.184583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-96ee1e4d-5a96-45c2-a188-20f3dd840227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:50.264828 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1188/2381] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:50.271375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.291581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:50.335932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90719917-7343-4184-87f9-86f668695d6a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.340587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.341185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.341460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.341730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.342363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.344292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.371032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1194/2382] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:50 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:50.375102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ee1e4d-5a96-45c2-a188-20f3dd840227 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.385937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ee1e4d-5a96-45c2-a188-20f3dd840227 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.442586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.443209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.443487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.443759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.444379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.446287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.466606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1189/2383] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:50 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:50.470660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.481841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287f5c2-0940-4479-b8d2-6076b887946b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.486434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.487060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.487342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.487615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.488251 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.490034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.500554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1195/2384] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:50.504413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.514899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e719ad-529a-4a0f-b37c-db40b5ff9b87 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.519417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58dc18d-775a-45df-8bdc-23c561940f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.519997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58dc18d-775a-45df-8bdc-23c561940f7e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.520271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58dc18d-775a-45df-8bdc-23c561940f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.520695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58dc18d-775a-45df-8bdc-23c561940f7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.521411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58dc18d-775a-45df-8bdc-23c561940f7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.522993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58dc18d-775a-45df-8bdc-23c561940f7e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.532641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1190/2385] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:50.537231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.547182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da61392f-810d-499a-bcb1-dca48cad0443 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.551635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.552349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.552620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.552889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] PATH_INFO: `/v3/projects/a2ecd72df0bb4eb1b4ed3ec9e99a685f/users/876025fcedcf41ad9b60b8c559b97647/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.566873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2ecd72df0bb4eb1b4ed3ec9e99a685f, user_id=876025fcedcf41ad9b60b8c559b97647, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.570203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.583361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1196/2386] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:50 2022] PUT /identity/v3/projects/a2ecd72df0bb4eb1b4ed3ec9e99a685f/users/876025fcedcf41ad9b60b8c559b97647/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:50.593346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6059f128-f88c-4da0-9cc1-ee6391759a4a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.593631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6059f128-f88c-4da0-9cc1-ee6391759a4a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.593901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6059f128-f88c-4da0-9cc1-ee6391759a4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.623816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6059f128-f88c-4da0-9cc1-ee6391759a4a None None] MFA Rules not processed for user `876025fcedcf41ad9b60b8c559b97647`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:50.625005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6059f128-f88c-4da0-9cc1-ee6391759a4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:50.691736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1191/2387] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:50.697694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.761038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c59452-1cf7-4b3f-8db5-1e6a76ad1ed2 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.765715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.766311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.766592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.766866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.767530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.769455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.796950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1197/2388] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:50 2022] POST /identity/v3/projects => generated 385 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:50.801074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6059f128-f88c-4da0-9cc1-ee6391759a4a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.811712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6059f128-f88c-4da0-9cc1-ee6391759a4a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.816440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:50.868568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.869409 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.869687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.869958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.870575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.872515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.893189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1192/2389] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:50 2022] POST /identity/v3/users => generated 299 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:50.897725 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.907846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d1277d-b223-4476-a2b2-4ac97a0883aa admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.913077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.913668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.913949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.914224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.914864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.916579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.927133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1198/2390] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:50.931329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.940992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bac57f-5da4-45e7-aa69-930082118573 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:50.945619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:50.946229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:50.946502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:50.946853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] PATH_INFO: `/v3/projects/0c67563b4c8249328d562623c0b6e786/users/0a7e184581f54e14a0bcbe364e978a5a/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:50.962128 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c67563b4c8249328d562623c0b6e786, user_id=0a7e184581f54e14a0bcbe364e978a5a, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:50.964888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:50.980997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1193/2391] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:50 2022] PUT /identity/v3/projects/0c67563b4c8249328d562623c0b6e786/users/0a7e184581f54e14a0bcbe364e978a5a/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:50.985117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:50.995984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfcc8d2-cfbd-4017-b6e9-aad32e6fbd57 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:51.037213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:51.054931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62ae781-8ad7-4a59-8a45-e6da9405ff39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:51.055503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ae781-8ad7-4a59-8a45-e6da9405ff39 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:51.055782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ae781-8ad7-4a59-8a45-e6da9405ff39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:51.056060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ae781-8ad7-4a59-8a45-e6da9405ff39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:51.056696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62ae781-8ad7-4a59-8a45-e6da9405ff39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:51.058296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62ae781-8ad7-4a59-8a45-e6da9405ff39 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:51.068490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1199/2392] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:40:50 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:40:51.072669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:51.083673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e005cabe-703e-4715-80b9-a034a1ea380a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:51.088315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:51.088945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:51.089219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:51.089487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] PATH_INFO: `/v3/projects/0c67563b4c8249328d562623c0b6e786/users/0a7e184581f54e14a0bcbe364e978a5a/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:51.091390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c67563b4c8249328d562623c0b6e786, user_id=0a7e184581f54e14a0bcbe364e978a5a, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:51.093987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:51.104034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1194/2393] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:40:51 2022] PUT /identity/v3/projects/0c67563b4c8249328d562623c0b6e786/users/0a7e184581f54e14a0bcbe364e978a5a/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:51.114064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:51.114370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:51.114667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:51.142621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] MFA Rules not processed for user `0a7e184581f54e14a0bcbe364e978a5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:51.143839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:51.212016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1200/2394] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:51 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:51.223163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:51.224859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:51.283562 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf47054-27f7-465b-b186-c6df9c07630c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:51.287358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:51.288080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:51.288233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:51.288506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:51.355595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:51.357306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:51.367435 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1195/2395] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:52.877787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:52.879218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:52.891663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd2766b-ea16-4f29-899f-cb158fcf031c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:52.895188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:52.896411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:52.896683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:52.896948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:52.957432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:52.959621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:52.971634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1201/2396] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:53.503937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:53.506540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:53.541177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:53.601234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9f70bf-dd11-408e-97ff-5761a626077a service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:53.608049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd574b5-7655-4b63-a5a1-bafd10f8c8c6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:53.608873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd574b5-7655-4b63-a5a1-bafd10f8c8c6 service placement] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:53.609323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd574b5-7655-4b63-a5a1-bafd10f8c8c6 service placement] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:53.609738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd574b5-7655-4b63-a5a1-bafd10f8c8c6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:53.612936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-efd574b5-7655-4b63-a5a1-bafd10f8c8c6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:53.626402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd574b5-7655-4b63-a5a1-bafd10f8c8c6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:53.628667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd574b5-7655-4b63-a5a1-bafd10f8c8c6 service placement] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:53.641432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1196/2397] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:53.658385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:53.668839 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7094ef-35f4-49b3-a599-7f231b791abd service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:53.674845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:53.675493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:53.675770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:53.676042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:53.686878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:53.688595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:53.727787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1202/2398] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:53 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:56.741559 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9a855-4460-4173-8423-0a4222f51187 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:56.742097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9a855-4460-4173-8423-0a4222f51187 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:56.742097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9a855-4460-4173-8423-0a4222f51187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:56.761690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-faa9a855-4460-4173-8423-0a4222f51187 None None] MFA Rules not processed for user `f68d73b5efc74b3380429cf72e506905`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:56.762881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-faa9a855-4460-4173-8423-0a4222f51187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:56.818697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1197/2399] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:57.675094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:57.727269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:57.743205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fae8cc-defd-4acd-bad9-dbf3ccedb75b service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:57.748163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:57.748907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:57.749191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:57.749465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] PATH_INFO: `/v3/users/3c66003363624d3bb09a7b9be6e6193f` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:57.750760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c66003363624d3bb09a7b9be6e6193f)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:57.753001 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:57.805495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c66003363624d3bb09a7b9be6e6193f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:57.806090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c66003363624d3bb09a7b9be6e6193f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:57.812964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c66003363624d3bb09a7b9be6e6193f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:57.813592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c66003363624d3bb09a7b9be6e6193f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:57.827462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c66003363624d3bb09a7b9be6e6193f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:57.841495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1203/2400] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:57 2022] DELETE /identity/v3/users/3c66003363624d3bb09a7b9be6e6193f => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:57.845913 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:57.846387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:57.846503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:57.866417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:57.867687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:57.934075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1198/2401] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:57.944007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:57.945429 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:58.005274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652a0d14-de2c-4e36-80de-6824334221b6 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:58.009189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:58.009795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:58.010070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:58.010338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:58.021900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:58.025802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:58.038341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1204/2402] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:40:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:40:58.313124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:58.314579 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:58.327938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d614f9ea-dd6f-4973-9105-7e47d471eb3f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:58.332653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:58.333882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:58.334194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:58.334572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] PATH_INFO: `/v3/projects/42f57370bf3a4d3fb7499005ff555cba` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:58.335731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] RBAC: Authorizing `identity:delete_project(project_id=42f57370bf3a4d3fb7499005ff555cba)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:58.337724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:58.382177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:58.385522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42f57370bf3a4d3fb7499005ff555cba', 'request_id': 'req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.392561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42f57370bf3a4d3fb7499005ff555cba', 'request_id': 'req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.398603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42f57370bf3a4d3fb7499005ff555cba', 'request_id': 'req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.411806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] The token cache is being invalidate because project 42f57370bf3a4d3fb7499005ff555cba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:58.412097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.412726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.414204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1199/2403] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:58 2022] DELETE /identity/v3/projects/42f57370bf3a4d3fb7499005ff555cba => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:58.418189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:58.428410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19ebcff-7ac3-44cd-a4a9-c18597935c39 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:58.487268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:58.487890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:58.488165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:58.488435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] PATH_INFO: `/v3/users/876025fcedcf41ad9b60b8c559b97647` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:58.489797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=876025fcedcf41ad9b60b8c559b97647)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:58.491831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:58.504507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:58.546662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '876025fcedcf41ad9b60b8c559b97647'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.547497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '876025fcedcf41ad9b60b8c559b97647'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.556841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '876025fcedcf41ad9b60b8c559b97647'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.557724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '876025fcedcf41ad9b60b8c559b97647'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.571667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '876025fcedcf41ad9b60b8c559b97647'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:58.595055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1205/2404] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:58 2022] DELETE /identity/v3/users/876025fcedcf41ad9b60b8c559b97647 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:58.891600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:58.893052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:58.907131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c4dfe5-4224-47e6-8c13-b26f2d1a4c3b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:58.966877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:58.967548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:58.967823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:58.968093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] PATH_INFO: `/v3/projects/a2ecd72df0bb4eb1b4ed3ec9e99a685f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:58.969191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2ecd72df0bb4eb1b4ed3ec9e99a685f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:58.971119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:59.022861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2ecd72df0bb4eb1b4ed3ec9e99a685f', 'request_id': 'req-36c78154-60f0-42fd-a019-962dab6ed422'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.029577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2ecd72df0bb4eb1b4ed3ec9e99a685f', 'request_id': 'req-36c78154-60f0-42fd-a019-962dab6ed422'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.035406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2ecd72df0bb4eb1b4ed3ec9e99a685f', 'request_id': 'req-36c78154-60f0-42fd-a019-962dab6ed422'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.049329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] The token cache is being invalidate because project a2ecd72df0bb4eb1b4ed3ec9e99a685f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:59.049743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.050387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.052882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1200/2405] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:58 2022] DELETE /identity/v3/projects/a2ecd72df0bb4eb1b4ed3ec9e99a685f => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:59.057000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:59.058427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:59.072328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5a77a1-5a9a-4c99-a54f-820cbec3f4bb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:59.131443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:59.131982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:59.132260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:59.132532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] PATH_INFO: `/v3/users/0a7e184581f54e14a0bcbe364e978a5a` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:59.133667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a7e184581f54e14a0bcbe364e978a5a)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:59.135633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:59.183473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a7e184581f54e14a0bcbe364e978a5a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.184250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a7e184581f54e14a0bcbe364e978a5a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.193760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a7e184581f54e14a0bcbe364e978a5a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.194750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a7e184581f54e14a0bcbe364e978a5a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.208964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c3bd63-49b8-4dd2-8cc6-7508fcb0795f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a7e184581f54e14a0bcbe364e978a5a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.225760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1206/2406] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:40:59 2022] DELETE /identity/v3/users/0a7e184581f54e14a0bcbe364e978a5a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:40:59.531355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:59.533051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:59.547554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c78154-60f0-42fd-a019-962dab6ed422 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:59.619402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:59.620061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:59.620321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:59.620591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] PATH_INFO: `/v3/projects/0c67563b4c8249328d562623c0b6e786` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:59.621719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c67563b4c8249328d562623c0b6e786)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:59.623910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:59.631866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:59.685452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c67563b4c8249328d562623c0b6e786', 'request_id': 'req-f95cc1ca-8095-482b-81ae-62fdf3d57f24'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.691989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c67563b4c8249328d562623c0b6e786', 'request_id': 'req-f95cc1ca-8095-482b-81ae-62fdf3d57f24'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.698526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c67563b4c8249328d562623c0b6e786', 'request_id': 'req-f95cc1ca-8095-482b-81ae-62fdf3d57f24'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.712441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] The token cache is being invalidate because project 0c67563b4c8249328d562623c0b6e786 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:40:59.712725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.713351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-f95cc1ca-8095-482b-81ae-62fdf3d57f24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:40:59.714852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1201/2407] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:40:59 2022] DELETE /identity/v3/projects/0c67563b4c8249328d562623c0b6e786 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:40:59.719913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f0e81a-34e1-4766-9fe7-1dbdfd37346f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:59.720201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f0e81a-34e1-4766-9fe7-1dbdfd37346f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:59.720474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f0e81a-34e1-4766-9fe7-1dbdfd37346f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:59.741851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a8f0e81a-34e1-4766-9fe7-1dbdfd37346f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:59.743202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a8f0e81a-34e1-4766-9fe7-1dbdfd37346f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:59.748029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:40:59.807989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1207/2408] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:40:59.815648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2b2ec5-c0a2-480a-a672-fea08fab8437 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:59.815953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2b2ec5-c0a2-480a-a672-fea08fab8437 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:59.816243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2b2ec5-c0a2-480a-a672-fea08fab8437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:59.836543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8b2b2ec5-c0a2-480a-a672-fea08fab8437 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:40:59.837742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8b2b2ec5-c0a2-480a-a672-fea08fab8437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:40:59.846474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1202/2409] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:40:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:40:59.850806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f0e81a-34e1-4766-9fe7-1dbdfd37346f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:59.860500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f0e81a-34e1-4766-9fe7-1dbdfd37346f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:59.865835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:59.866504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:59.866816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:59.867152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:59.867831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:40:59.868127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:59.869980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:59.882868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1208/2410] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:40:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:40:59.887202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2b2ec5-c0a2-480a-a672-fea08fab8437 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:59.896652 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2b2ec5-c0a2-480a-a672-fea08fab8437 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:40:59.901350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:40:59.902017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:40:59.902354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:40:59.902688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:40:59.903442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:40:59.905397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:40:59.939478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1203/2411] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:40:59 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:40:59.943829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:40:59.954089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d767de9-f072-46e2-b820-d5559ddb9097 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.013482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.014089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.014368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.014640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.015304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.017236 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.041843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1209/2412] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:40:59 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:41:00.046135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.056491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a23af3-5c48-46f7-87b7-6e955cd6198a admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.061196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.061765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.062043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.062309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.062944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.064581 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.076077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1204/2413] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:41:00.080094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.092124 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feda40e-69a3-4d72-95c9-3ecf57f4652b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.097759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc2ace0-ca23-47d9-9a2b-be88acf46493 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.098365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc2ace0-ca23-47d9-9a2b-be88acf46493 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.098644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc2ace0-ca23-47d9-9a2b-be88acf46493 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.099559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc2ace0-ca23-47d9-9a2b-be88acf46493 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.100151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc2ace0-ca23-47d9-9a2b-be88acf46493 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.101788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc2ace0-ca23-47d9-9a2b-be88acf46493 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.112957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1210/2414] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:41:00.116960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.127468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5de1c3c-1da1-42e2-9366-d8e8677a1574 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.132797 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.133444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.133723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.134000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] PATH_INFO: `/v3/projects/68378dce4de84f00b8f0cec1468b9473/users/5a65a96339c04416b5a7afebbb8a69c8/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.151100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] RBAC: Authorizing `identity:create_grant(project_id=68378dce4de84f00b8f0cec1468b9473, user_id=5a65a96339c04416b5a7afebbb8a69c8, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.153892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.168169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1205/2415] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:41:00 2022] PUT /identity/v3/projects/68378dce4de84f00b8f0cec1468b9473/users/5a65a96339c04416b5a7afebbb8a69c8/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:00.236849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861066ad-48b6-44f1-8131-07efd463118a None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.237136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861066ad-48b6-44f1-8131-07efd463118a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.237467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861066ad-48b6-44f1-8131-07efd463118a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.266457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-861066ad-48b6-44f1-8131-07efd463118a None None] MFA Rules not processed for user `5a65a96339c04416b5a7afebbb8a69c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:00.267717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-861066ad-48b6-44f1-8131-07efd463118a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:00.346502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1211/2416] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:00 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:41:00.350885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.416123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6bc869-cce4-4dc8-956a-24d3e82c1846 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.420786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.421384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.421657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.421925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.422548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.424456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.455146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1206/2417] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:41:00 2022] POST /identity/v3/projects => generated 383 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:41:00.456274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861066ad-48b6-44f1-8131-07efd463118a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.463309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:00.467408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861066ad-48b6-44f1-8131-07efd463118a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.526486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.527122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.527401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.527672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.528310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.530264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.555778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1212/2418] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:41:00 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:41:00.557013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.567805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f83465-70ce-4290-8819-66909476b4d2 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.572918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.573516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.573808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.574077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.574778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.576400 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.596526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1207/2419] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:00 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:41:00.601205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.611346 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b27fbf9-4731-4fd7-8206-98fd955d1add admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.616148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de04eecf-b390-4e87-890f-d2ef4bd2628b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.616749 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de04eecf-b390-4e87-890f-d2ef4bd2628b admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.617025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de04eecf-b390-4e87-890f-d2ef4bd2628b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.617357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de04eecf-b390-4e87-890f-d2ef4bd2628b admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.617997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de04eecf-b390-4e87-890f-d2ef4bd2628b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.619930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de04eecf-b390-4e87-890f-d2ef4bd2628b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.630352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1213/2420] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:41:00.634607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.641355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:00.646115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dca2b05-f10a-4da3-9acb-4823aad776a2 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.658855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.659742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.660082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.660519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] PATH_INFO: `/v3/projects/3c31bb107a874a47b2bdb8b89cec80c2/users/c055031d571744c6988c1ea9d3f7a366/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.676234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c31bb107a874a47b2bdb8b89cec80c2, user_id=c055031d571744c6988c1ea9d3f7a366, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.679511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.695846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1208/2421] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:41:00 2022] PUT /identity/v3/projects/3c31bb107a874a47b2bdb8b89cec80c2/users/c055031d571744c6988c1ea9d3f7a366/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:00.706106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1b535-653e-416d-8e79-082902e6170f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.706391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1b535-653e-416d-8e79-082902e6170f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.706665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1b535-653e-416d-8e79-082902e6170f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.736588 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ffc1b535-653e-416d-8e79-082902e6170f None None] MFA Rules not processed for user `c055031d571744c6988c1ea9d3f7a366`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:00.737939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ffc1b535-653e-416d-8e79-082902e6170f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:00.802411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1214/2422] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:00 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:41:00.806832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.869860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d209cc-17f1-4268-a6d6-75e5410631fc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.874532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.875164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.875442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.875709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.876341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.878217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:00.905455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1209/2423] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:41:00 2022] POST /identity/v3/projects => generated 385 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:41:00.909497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1b535-653e-416d-8e79-082902e6170f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:00.920270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1b535-653e-416d-8e79-082902e6170f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:00.976541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:00.977149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:00.977422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:00.977690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:00.978317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:00.980269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:01.003277 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1215/2424] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:41:00 2022] POST /identity/v3/users => generated 299 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:41:01.008418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:01.018823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44d9f10-46db-4b3f-a48c-7658f8dea8fc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:01.024610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:01.025210 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:01.025482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:01.025778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:01.026419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:01.028072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:01.044225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1210/2425] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:01 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:41:01.048526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:01.053992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:01.060232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d3b79a-3f71-4b93-a2b3-af28a8857301 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:01.065282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:01.065943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:01.066221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:01.066497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] PATH_INFO: `/v3/projects/f05ee7d782f84cd7a9ab00670bef40f3/users/2e5e6b91549b4f6c800d1b0a8fe14e88/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:01.084115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f05ee7d782f84cd7a9ab00670bef40f3, user_id=2e5e6b91549b4f6c800d1b0a8fe14e88, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:01.086746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:01.101545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1216/2426] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:41:01 2022] PUT /identity/v3/projects/f05ee7d782f84cd7a9ab00670bef40f3/users/2e5e6b91549b4f6c800d1b0a8fe14e88/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:01.105728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:01.115959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2879804a-5b6a-4d40-9a09-684e0e2ea509 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:01.172487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9679f580-df23-4824-8f31-000851ceef7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:01.173095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9679f580-df23-4824-8f31-000851ceef7e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:01.173352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9679f580-df23-4824-8f31-000851ceef7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:01.173620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9679f580-df23-4824-8f31-000851ceef7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:01.174266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9679f580-df23-4824-8f31-000851ceef7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:01.175874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9679f580-df23-4824-8f31-000851ceef7e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:01.185754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1211/2427] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:01 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:41:01.189820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:01.200593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a182be-7598-4931-b11b-5e78b957ca5a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:01.205156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:01.205789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:01.206065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:01.206336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] PATH_INFO: `/v3/projects/f05ee7d782f84cd7a9ab00670bef40f3/users/2e5e6b91549b4f6c800d1b0a8fe14e88/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:01.208325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f05ee7d782f84cd7a9ab00670bef40f3, user_id=2e5e6b91549b4f6c800d1b0a8fe14e88, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:01.211004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:01.221554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1217/2428] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:41:01 2022] PUT /identity/v3/projects/f05ee7d782f84cd7a9ab00670bef40f3/users/2e5e6b91549b4f6c800d1b0a8fe14e88/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:01.231757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:01.232041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:01.232310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:01.261810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] MFA Rules not processed for user `2e5e6b91549b4f6c800d1b0a8fe14e88`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:01.263081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:01.301245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:01.331028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1212/2429] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:01 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:41:01.342584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:01.344285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:01.408934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92e181-43b5-4443-ba64-993b0af27d58 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:01.412712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:01.413304 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:01.413577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:01.413843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:01.469754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:01.471138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:01.480000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1218/2430] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:41:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:41:02.993097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:02.994751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:03.008558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a2cd01-d618-4dce-81f3-033a6158e58f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:03.012172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:03.012759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:03.013036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:03.013307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:03.067978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:03.069385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:03.078080 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1213/2431] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:41:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:41:03.644107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:03.653090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7042461e-f764-42c6-a20e-83f609e36c7e service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:03.656719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95221780-5f6c-4cd1-b775-d56f6cb4725b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:03.657311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95221780-5f6c-4cd1-b775-d56f6cb4725b service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:03.657586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95221780-5f6c-4cd1-b775-d56f6cb4725b service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:03.657856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95221780-5f6c-4cd1-b775-d56f6cb4725b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:03.667521 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95221780-5f6c-4cd1-b775-d56f6cb4725b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:03.668949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95221780-5f6c-4cd1-b775-d56f6cb4725b service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:03.678698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1219/2432] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:41:03 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:41:07.625680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:07.688565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f2d9ba-f582-406c-8b94-11dfd89b7ccd service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:07.693641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:07.694254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:07.694533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:07.694802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] PATH_INFO: `/v3/users/5a65a96339c04416b5a7afebbb8a69c8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:07.695936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a65a96339c04416b5a7afebbb8a69c8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:07.697843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:07.744389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a65a96339c04416b5a7afebbb8a69c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:07.744965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a65a96339c04416b5a7afebbb8a69c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:07.752071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a65a96339c04416b5a7afebbb8a69c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:07.767622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a65a96339c04416b5a7afebbb8a69c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:07.768024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a65a96339c04416b5a7afebbb8a69c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:07.771413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:07.783796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1214/2433] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:41:07 2022] DELETE /identity/v3/users/5a65a96339c04416b5a7afebbb8a69c8 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:07.788707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:07.789327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:07.789327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:07.807951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:07.809130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:07.866853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1220/2434] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:41:07.876809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:07.878220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:07.936197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4aaa67-f09e-43a2-a9ab-ba01b16a4348 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:07.939836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:07.940431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:07.940707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:07.940977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:07.949879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:07.951325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:07.961737 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1215/2435] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:41:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:41:08.228593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:08.230106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:08.243720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f91dbd-da56-4e0e-976f-4b5c2cdf5a5d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:08.248778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:08.249415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:08.249693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:08.249971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] PATH_INFO: `/v3/projects/68378dce4de84f00b8f0cec1468b9473` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:08.256562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=68378dce4de84f00b8f0cec1468b9473)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:08.258525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:08.260297 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:08.307040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68378dce4de84f00b8f0cec1468b9473', 'request_id': 'req-283fee1a-86e4-4e0f-9e26-9b060b2421b5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.312336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68378dce4de84f00b8f0cec1468b9473', 'request_id': 'req-283fee1a-86e4-4e0f-9e26-9b060b2421b5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.319173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68378dce4de84f00b8f0cec1468b9473', 'request_id': 'req-283fee1a-86e4-4e0f-9e26-9b060b2421b5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.331307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] The token cache is being invalidate because project 68378dce4de84f00b8f0cec1468b9473 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:41:08.331591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.332203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.334257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1221/2436] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:41:08 2022] DELETE /identity/v3/projects/68378dce4de84f00b8f0cec1468b9473 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:08.338351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:08.347912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9344aa89-47a5-444e-8a42-9f65cac204d7 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:08.404506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:08.405135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:08.405411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:08.405683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] PATH_INFO: `/v3/users/c055031d571744c6988c1ea9d3f7a366` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:08.406803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] RBAC: Authorizing `identity:delete_user(user_id=c055031d571744c6988c1ea9d3f7a366)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:08.408816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:08.470446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c055031d571744c6988c1ea9d3f7a366'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.471602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c055031d571744c6988c1ea9d3f7a366'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.479144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c055031d571744c6988c1ea9d3f7a366'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.491178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c055031d571744c6988c1ea9d3f7a366'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.492028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c055031d571744c6988c1ea9d3f7a366'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.505195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1216/2437] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:41:08 2022] DELETE /identity/v3/users/c055031d571744c6988c1ea9d3f7a366 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:08.813011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:08.815361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:08.843433 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fee1a-86e4-4e0f-9e26-9b060b2421b5 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:08.904379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:08.905020 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:08.905292 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:08.905561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] PATH_INFO: `/v3/projects/3c31bb107a874a47b2bdb8b89cec80c2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:08.906873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c31bb107a874a47b2bdb8b89cec80c2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:08.909361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:08.957871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c31bb107a874a47b2bdb8b89cec80c2', 'request_id': 'req-76aa6c98-d146-437d-a34f-f493dabfd04c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.963814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c31bb107a874a47b2bdb8b89cec80c2', 'request_id': 'req-76aa6c98-d146-437d-a34f-f493dabfd04c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.970134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c31bb107a874a47b2bdb8b89cec80c2', 'request_id': 'req-76aa6c98-d146-437d-a34f-f493dabfd04c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.984982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] The token cache is being invalidate because project 3c31bb107a874a47b2bdb8b89cec80c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:41:08.985309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.985938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:08.987517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1222/2438] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:41:08 2022] DELETE /identity/v3/projects/3c31bb107a874a47b2bdb8b89cec80c2 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:08.992120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:08.993545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:09.009013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02d631e-7659-4c5a-8786-48dd90a12deb admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:09.021999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:09.073572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:09.074196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.074471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.074764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] PATH_INFO: `/v3/users/2e5e6b91549b4f6c800d1b0a8fe14e88` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.075962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e5e6b91549b4f6c800d1b0a8fe14e88)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:09.077976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:09.126569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e5e6b91549b4f6c800d1b0a8fe14e88'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.127253 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e5e6b91549b4f6c800d1b0a8fe14e88'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.134402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e5e6b91549b4f6c800d1b0a8fe14e88'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.147082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e5e6b91549b4f6c800d1b0a8fe14e88'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.147780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cf080795-53b3-4889-95e1-dce22489bb4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e5e6b91549b4f6c800d1b0a8fe14e88'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.161459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1217/2439] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:41:08 2022] DELETE /identity/v3/users/2e5e6b91549b4f6c800d1b0a8fe14e88 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:09.459107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:09.460697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:09.475498 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa6c98-d146-437d-a34f-f493dabfd04c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:09.495707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:09.539539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:09.540174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.540450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.540724 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] PATH_INFO: `/v3/projects/f05ee7d782f84cd7a9ab00670bef40f3` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.541837 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f05ee7d782f84cd7a9ab00670bef40f3)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:09.543786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:09.593004 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f05ee7d782f84cd7a9ab00670bef40f3', 'request_id': 'req-3e09496b-1dd2-4260-8a0b-4968b5f549f3'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.598606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f05ee7d782f84cd7a9ab00670bef40f3', 'request_id': 'req-3e09496b-1dd2-4260-8a0b-4968b5f549f3'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.605113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f05ee7d782f84cd7a9ab00670bef40f3', 'request_id': 'req-3e09496b-1dd2-4260-8a0b-4968b5f549f3'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.618076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] The token cache is being invalidate because project f05ee7d782f84cd7a9ab00670bef40f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:41:09.618385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.619362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3e09496b-1dd2-4260-8a0b-4968b5f549f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:09.620948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1223/2440] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:41:09 2022] DELETE /identity/v3/projects/f05ee7d782f84cd7a9ab00670bef40f3 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:09.625779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124cdaa3-7a2d-4471-b8d0-5f69e8f2e452 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.626064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124cdaa3-7a2d-4471-b8d0-5f69e8f2e452 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.626336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124cdaa3-7a2d-4471-b8d0-5f69e8f2e452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.647926 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-124cdaa3-7a2d-4471-b8d0-5f69e8f2e452 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:09.649219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-124cdaa3-7a2d-4471-b8d0-5f69e8f2e452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:09.711649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1218/2441] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:41:09.719540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219a66ba-6b9f-4113-ae2a-b3d5dadd9878 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.719824 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219a66ba-6b9f-4113-ae2a-b3d5dadd9878 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.720101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219a66ba-6b9f-4113-ae2a-b3d5dadd9878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.740656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-219a66ba-6b9f-4113-ae2a-b3d5dadd9878 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:09.741864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-219a66ba-6b9f-4113-ae2a-b3d5dadd9878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:09.750021 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1224/2442] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:41:09.754296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124cdaa3-7a2d-4471-b8d0-5f69e8f2e452 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:09.764557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124cdaa3-7a2d-4471-b8d0-5f69e8f2e452 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:09.769213 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:09.769749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.770064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.770505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.771308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:41:09.771613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:09.773216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:09.785031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1219/2443] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:41:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:41:09.789494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219a66ba-6b9f-4113-ae2a-b3d5dadd9878 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:09.799050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219a66ba-6b9f-4113-ae2a-b3d5dadd9878 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:09.804419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:09.805079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.805419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.805752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.806445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:09.808468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:09.837653 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1225/2444] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:41:09 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:41:09.841855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:09.852501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a3e9a2-24d8-4807-bc2a-f6f8bb9be6fa admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:09.871807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:09.931564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:09.932366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.932805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.933234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.934003 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:09.936272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:09.961460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1220/2445] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:41:09 2022] POST /identity/v3/users => generated 285 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:41:09.965494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:09.976917 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6bd4ce-f9ee-4a25-be18-a80e5de823eb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:09.981599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:09.982259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:09.982594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:09.982922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:09.983661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:09.985259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:09.996027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1226/2446] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:41:10.000083 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:10.013770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c40355-03b9-4c7b-b922-91629b03e3f1 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:10.018838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254819a6-cac7-4d86-8331-f13a3a5d69fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:10.019586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254819a6-cac7-4d86-8331-f13a3a5d69fb admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:10.019985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254819a6-cac7-4d86-8331-f13a3a5d69fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:10.020331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254819a6-cac7-4d86-8331-f13a3a5d69fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:10.021205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254819a6-cac7-4d86-8331-f13a3a5d69fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:10.022874 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254819a6-cac7-4d86-8331-f13a3a5d69fb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:10.034137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1221/2447] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:41:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:41:10.038997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:10.050936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a832cd42-0a75-4b52-a98e-83ae9755ec67 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:10.057010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:10.058130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:10.058130 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:10.058340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] PATH_INFO: `/v3/projects/2c69adb82e8f4e75860f0d712554eb71/users/f2f2115a44c14b47b7236afe87ee1853/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:10.075419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c69adb82e8f4e75860f0d712554eb71, user_id=f2f2115a44c14b47b7236afe87ee1853, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:10.078484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:10.106560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1227/2448] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:41:10 2022] PUT /identity/v3/projects/2c69adb82e8f4e75860f0d712554eb71/users/f2f2115a44c14b47b7236afe87ee1853/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:10.108526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2951420-9167-4b45-93f1-8e51b2ab5f2d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:10.109625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2951420-9167-4b45-93f1-8e51b2ab5f2d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:10.109999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2951420-9167-4b45-93f1-8e51b2ab5f2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:10.141767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f2951420-9167-4b45-93f1-8e51b2ab5f2d None None] MFA Rules not processed for user `f2f2115a44c14b47b7236afe87ee1853`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:10.143036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f2951420-9167-4b45-93f1-8e51b2ab5f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:10.220372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1222/2449] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:10 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:41:10.232507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:10.234532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:10.296357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:10.304741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e43f749-22c2-4c7a-a669-ef3408703377 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:10.309345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b34c61ebdb9147d5b3d66cbfa556f38e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:10.310145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:10.310619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:10.310889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:10.320111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:10.321962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:10.330931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1228/2450] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:41:10 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:41:20.215028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=56,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:20.326700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:39.805470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:39.805934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:39.806033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:39.825375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:39.826658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:39.888340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1223/2451] 10.222.0.33 () {58 vars in 967 bytes} [Thu Oct 6 04:41:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:41:39.937822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:39.939241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:40.010136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17685cbd-6f64-4c8e-a77d-3cf107d548e0 service glance] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:40.020569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:40.021173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:40.021450 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:40.021717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:40.031078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:40.032652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:40.040682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1229/2452] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:41:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:41:46.858522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:46.860105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:46.939165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd716047-8a04-4ed3-aef5-5895a1980da0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:46.943332 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:46.944863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:46.944977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:46.945545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:46.945822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:46.946092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:46.948688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:46.961838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:46.964606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:46.979191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1224/2453] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:41:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:41:47.038287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d7eb97-4c5a-483a-9fe0-95f53b60445c service swift] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:47.043776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:47.044413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:47.044694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:47.044965 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] PATH_INFO: `/v3/users/ddfa371a39b84f42bba4621ae3a99794` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:47.046248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddfa371a39b84f42bba4621ae3a99794)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:47.048186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:47.098025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddfa371a39b84f42bba4621ae3a99794'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.098782 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddfa371a39b84f42bba4621ae3a99794'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.105637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddfa371a39b84f42bba4621ae3a99794'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.106252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddfa371a39b84f42bba4621ae3a99794'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.118117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddfa371a39b84f42bba4621ae3a99794'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.131758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1230/2454] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:41:46 2022] DELETE /identity/v3/users/ddfa371a39b84f42bba4621ae3a99794 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:41:47.306506 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:47.319278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc620b9d-9a62-4ba3-90f7-5fc2688cb3cf service placement] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:47.377181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:47.377800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:47.378077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:47.378407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] PATH_INFO: `/v3/projects/43ee2f7c65044158a0230e4997d7bdf9` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:47.379551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43ee2f7c65044158a0230e4997d7bdf9)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:47.381441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:47.413767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:47.427629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43ee2f7c65044158a0230e4997d7bdf9', 'request_id': 'req-4502d8c1-79a3-48c1-a237-785fef767519'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.434271 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43ee2f7c65044158a0230e4997d7bdf9', 'request_id': 'req-4502d8c1-79a3-48c1-a237-785fef767519'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.439966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43ee2f7c65044158a0230e4997d7bdf9', 'request_id': 'req-4502d8c1-79a3-48c1-a237-785fef767519'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.459092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] The token cache is being invalidate because project 43ee2f7c65044158a0230e4997d7bdf9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:41:47.459385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.460018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.461521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1225/2455] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:41:47 2022] DELETE /identity/v3/projects/43ee2f7c65044158a0230e4997d7bdf9 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:47.473946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:47.475384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:47.493025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583da8f9-4a25-404f-a91e-29e0b2d5cab2 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:47.498592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:47.553912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64276590-4124-4706-bc79-69cee195104c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:47.554459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64276590-4124-4706-bc79-69cee195104c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:47.554738 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64276590-4124-4706-bc79-69cee195104c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:47.555035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64276590-4124-4706-bc79-69cee195104c admin admin] PATH_INFO: `/v3/users/9ebdc312cb65479ead36d13eef84ec5c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:47.556149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64276590-4124-4706-bc79-69cee195104c admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ebdc312cb65479ead36d13eef84ec5c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:47.558104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64276590-4124-4706-bc79-69cee195104c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:47.602874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ebdc312cb65479ead36d13eef84ec5c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.603537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ebdc312cb65479ead36d13eef84ec5c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.611046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ebdc312cb65479ead36d13eef84ec5c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.611671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ebdc312cb65479ead36d13eef84ec5c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.623625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ebdc312cb65479ead36d13eef84ec5c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.636057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1231/2456] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:41:47 2022] DELETE /identity/v3/users/9ebdc312cb65479ead36d13eef84ec5c => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:47.822943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:47.824536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:47.837286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4502d8c1-79a3-48c1-a237-785fef767519 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:47.896048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:47.896725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:47.896923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:47.897189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] PATH_INFO: `/v3/projects/d2b7266e22c2478e8e8b75229aae303e` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:47.898287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d2b7266e22c2478e8e8b75229aae303e)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:47.900185 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:47.946217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd2b7266e22c2478e8e8b75229aae303e', 'request_id': 'req-c3158ab3-a7f3-435e-b889-6de749c5395a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.952641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd2b7266e22c2478e8e8b75229aae303e', 'request_id': 'req-c3158ab3-a7f3-435e-b889-6de749c5395a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.958036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd2b7266e22c2478e8e8b75229aae303e', 'request_id': 'req-c3158ab3-a7f3-435e-b889-6de749c5395a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.970307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] The token cache is being invalidate because project d2b7266e22c2478e8e8b75229aae303e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:41:47.970615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.971259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c3158ab3-a7f3-435e-b889-6de749c5395a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:47.972715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1226/2457] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:41:47 2022] DELETE /identity/v3/projects/d2b7266e22c2478e8e8b75229aae303e => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:41:54.494493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:54.495907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:54.508761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64276590-4124-4706-bc79-69cee195104c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:54.570599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:54.571256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:54.571536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:54.571808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] PATH_INFO: `/v3/users/f2f2115a44c14b47b7236afe87ee1853` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:54.572992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2f2115a44c14b47b7236afe87ee1853)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:54.574982 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:54.607873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:54.622566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2f2115a44c14b47b7236afe87ee1853'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:54.623351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2f2115a44c14b47b7236afe87ee1853'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:54.630169 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2f2115a44c14b47b7236afe87ee1853'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:54.630793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2f2115a44c14b47b7236afe87ee1853'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:54.642869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2f2115a44c14b47b7236afe87ee1853'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:54.656032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1232/2458] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:41:54 2022] DELETE /identity/v3/users/f2f2115a44c14b47b7236afe87ee1853 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:41:54.660592 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:54.661137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:54.661245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:54.679962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:41:54.681136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:54.725108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:41:54.741332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1227/2459] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:41:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:41:54.751359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:54.752783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:54.808973 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1cebce-0e2a-4737-8bbb-8f01dd630842 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:54.813034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d2a89c-a9fe-4b1c-b6b9-8a1a7c31353b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:54.813633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d2a89c-a9fe-4b1c-b6b9-8a1a7c31353b service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:54.813909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d2a89c-a9fe-4b1c-b6b9-8a1a7c31353b service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:54.814180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d2a89c-a9fe-4b1c-b6b9-8a1a7c31353b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:54.823198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d2a89c-a9fe-4b1c-b6b9-8a1a7c31353b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:54.824605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d2a89c-a9fe-4b1c-b6b9-8a1a7c31353b service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:54.834189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1233/2460] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:41:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:41:55.104457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:41:55.105905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:41:55.118431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96622c7-a72f-4437-8b33-186a0778e99f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:41:55.123975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:41:55.124776 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:41:55.125057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:41:55.125331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] PATH_INFO: `/v3/projects/2c69adb82e8f4e75860f0d712554eb71` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:41:55.126456 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c69adb82e8f4e75860f0d712554eb71)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:41:55.128417 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:41:55.179627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c69adb82e8f4e75860f0d712554eb71', 'request_id': 'req-bf879300-cf98-4606-8a34-0a2f41a9edb7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:55.186317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c69adb82e8f4e75860f0d712554eb71', 'request_id': 'req-bf879300-cf98-4606-8a34-0a2f41a9edb7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:55.192096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c69adb82e8f4e75860f0d712554eb71', 'request_id': 'req-bf879300-cf98-4606-8a34-0a2f41a9edb7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:55.205132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] The token cache is being invalidate because project 2c69adb82e8f4e75860f0d712554eb71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:41:55.205419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:55.206082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:41:55.207619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1228/2461] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:41:55 2022] DELETE /identity/v3/projects/2c69adb82e8f4e75860f0d712554eb71 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:42:04.830131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=44,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:42:05.201378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=32,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:42:06.514821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:06.515310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:06.515402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:06.534970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:42:06.536182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:06.596438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1234/2462] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:42:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:42:06.608724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:42:06.610108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:06.680084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf879300-cf98-4606-8a34-0a2f41a9edb7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:42:06.685353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:42:06.685960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:06.686239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:06.686513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:06.694532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:42:06.696125 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:42:06.704841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1229/2463] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:42:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:42:10.646070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:42:10.647561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:10.709405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea2e038-efd5-4ce2-aae0-ef238ceb89f2 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:42:10.713565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca0b3f9-8404-4919-a40a-70a2c0cecaba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:42:10.714416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca0b3f9-8404-4919-a40a-70a2c0cecaba service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:10.714851 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca0b3f9-8404-4919-a40a-70a2c0cecaba service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:10.715195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca0b3f9-8404-4919-a40a-70a2c0cecaba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:10.718718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-aca0b3f9-8404-4919-a40a-70a2c0cecaba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:10.814941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca0b3f9-8404-4919-a40a-70a2c0cecaba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:42:10.816589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca0b3f9-8404-4919-a40a-70a2c0cecaba service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:42:10.839478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1235/2464] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:42:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:42:12.041107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:42:12.042936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:12.057201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19b7af0-ba16-40da-84ae-802b6b41b53d service swift] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:42:12.063066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:42:12.063694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:12.063971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:12.064242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:12.066072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:12.080628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:42:12.082094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:42:12.094474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1230/2465] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:42:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:42:20.836677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=88 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:42:22.088772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=38 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:42:22.211502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:22.211985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:22.212177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:22.231862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:42:22.233683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:22.240857 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1236/2466] 10.222.0.33 () {58 vars in 1009 bytes} [Thu Oct 6 04:42:22 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:42:22.253642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:42:22.265431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaec2e24-222c-4530-a195-7b4482095204 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:42:22.270290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e411d0a-fe32-4660-94b4-4c2cced8cd8e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:42:22.270870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e411d0a-fe32-4660-94b4-4c2cced8cd8e service swift] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:22.271188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e411d0a-fe32-4660-94b4-4c2cced8cd8e service swift] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:22.271459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e411d0a-fe32-4660-94b4-4c2cced8cd8e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:22.279476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e411d0a-fe32-4660-94b4-4c2cced8cd8e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:42:22.281033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e411d0a-fe32-4660-94b4-4c2cced8cd8e service swift] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:42:22.288837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1231/2467] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:42:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:42:27.578250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:42:27.579715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:27.596170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63134228-98fb-49f6-a6d2-df1027349d20 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:42:27.601635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:42:27.602239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:27.602514 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:27.602786 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:27.604549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:27.692803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:42:27.694719 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:42:27.716265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1237/2468] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:42:27 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:42:32.285728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=6 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:42:37.712794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=44 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:42:38.211107 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce878ba-61cc-4f1d-b676-7deb5731a0df None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:38.211611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce878ba-61cc-4f1d-b676-7deb5731a0df None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:38.211870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce878ba-61cc-4f1d-b676-7deb5731a0df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:38.263805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-dce878ba-61cc-4f1d-b676-7deb5731a0df None None] MFA Rules not processed for user `371190691e284514bafcba1bdd199fc6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:42:38.265110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dce878ba-61cc-4f1d-b676-7deb5731a0df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:42:38.285584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1232/2469] 10.222.0.33 () {58 vars in 967 bytes} [Thu Oct 6 04:42:38 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:42:38.298930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:42:38.320279 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71166721-97b0-467c-beca-d3c8cc8bf3d3 service nova] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:42:38.325499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2e280c4b418e4aafb822a617424c7e11', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:42:38.326089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:42:38.326360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:42:38.326655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:42:38.334727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:42:38.336321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:42:38.344311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1238/2470] 10.222.0.33 () {60 vars in 1384 bytes} [Thu Oct 6 04:42:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 06 04:42:48.281922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=18 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:42:48.341032 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=10 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:10.069415 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce878ba-61cc-4f1d-b676-7deb5731a0df None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:10.080104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce878ba-61cc-4f1d-b676-7deb5731a0df None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:10.083846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:10.084434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:10.084713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:10.084988 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:10.086713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:10.151122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:10.152536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:10.162674 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1233/2471] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:12.045955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:12.058107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67236870-8040-4c40-85f9-e2738fc05e59 service swift] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:12.061769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:12.062368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:12.063222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:12.063489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:12.065402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:12.078555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:12.080020 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:12.089977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1239/2472] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:13.435824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:13.437211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:13.455119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571d5b71-f772-47ec-bb57-8a862fd589b6 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:13.460306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:13.461190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:13.461632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:13.462161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] PATH_INFO: `/v3/users/f68d73b5efc74b3380429cf72e506905` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:13.479351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f68d73b5efc74b3380429cf72e506905)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:13.481804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:13.538305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f68d73b5efc74b3380429cf72e506905'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.539081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f68d73b5efc74b3380429cf72e506905'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.546656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f68d73b5efc74b3380429cf72e506905'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.559411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f68d73b5efc74b3380429cf72e506905'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.560140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-60257585-88f7-4ce0-bd04-74d0b0e5527e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f68d73b5efc74b3380429cf72e506905'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.574513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1234/2473] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:13 2022] DELETE /identity/v3/users/f68d73b5efc74b3380429cf72e506905 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:13.749436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:13.761406 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3893098c-f2d8-4998-96d2-c3d04e26f197 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:13.820643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:13.821262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:13.821533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:13.821804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] PATH_INFO: `/v3/projects/653e21693b6f4a3ca8b739f1290df3aa` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:13.834874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] RBAC: Authorizing `identity:delete_project(project_id=653e21693b6f4a3ca8b739f1290df3aa)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:13.840141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:13.897035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '653e21693b6f4a3ca8b739f1290df3aa', 'request_id': 'req-6570f277-0cbf-466f-ba73-3659743d7907'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.904137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '653e21693b6f4a3ca8b739f1290df3aa', 'request_id': 'req-6570f277-0cbf-466f-ba73-3659743d7907'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.913131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '653e21693b6f4a3ca8b739f1290df3aa', 'request_id': 'req-6570f277-0cbf-466f-ba73-3659743d7907'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.933807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] The token cache is being invalidate because project 653e21693b6f4a3ca8b739f1290df3aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:13.934114 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.934756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6570f277-0cbf-466f-ba73-3659743d7907 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:13.940661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1240/2474] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:13 2022] DELETE /identity/v3/projects/653e21693b6f4a3ca8b739f1290df3aa => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:13.942296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ce028-c9ca-4025-bc02-1c158a5a3cb4 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:13.942578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ce028-c9ca-4025-bc02-1c158a5a3cb4 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:13.942851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690ce028-c9ca-4025-bc02-1c158a5a3cb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:13.963609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-690ce028-c9ca-4025-bc02-1c158a5a3cb4 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:13.964806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-690ce028-c9ca-4025-bc02-1c158a5a3cb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:14.022311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:14.044108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1235/2475] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:14.052523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71383ac5-86ec-45e0-8853-af047874a8fd None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.052848 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71383ac5-86ec-45e0-8853-af047874a8fd None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.053134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71383ac5-86ec-45e0-8853-af047874a8fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.079578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-71383ac5-86ec-45e0-8853-af047874a8fd None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:14.080830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-71383ac5-86ec-45e0-8853-af047874a8fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:14.093228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1241/2476] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:14.094942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690ce028-c9ca-4025-bc02-1c158a5a3cb4 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:14.106281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690ce028-c9ca-4025-bc02-1c158a5a3cb4 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:14.111820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:14.112431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.112711 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.112999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.113643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:14.113946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:14.115586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:14.131009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1236/2477] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:14.135522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71383ac5-86ec-45e0-8853-af047874a8fd None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:14.145513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71383ac5-86ec-45e0-8853-af047874a8fd None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:14.151249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:14.152492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.152762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.153039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.153675 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:14.155700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:14.204603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1242/2478] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:14 2022] POST /identity/v3/projects => generated 365 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:14.205811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:14.219452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9eb0fd-f26c-4349-8c68-2309131800f4 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:14.307916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:14.308536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.308815 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.309181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.310195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:14.313148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:14.342604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1237/2479] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:14 2022] POST /identity/v3/users => generated 283 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:43:14.347989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:14.362896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42075998-b8bc-4eb9-a1be-f7a72bd87c57 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:14.368436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:14.369050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.369586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.369974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.370896 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:14.373707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:14.387505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1243/2480] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:14 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:14.391466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:14.404311 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b0aa69-cd65-40f9-b289-50f5be99f9b6 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:14.412370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25595ba7-7741-4c96-935f-25c753a764de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:14.412999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25595ba7-7741-4c96-935f-25c753a764de admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.413509 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25595ba7-7741-4c96-935f-25c753a764de admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.413996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25595ba7-7741-4c96-935f-25c753a764de admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.414879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25595ba7-7741-4c96-935f-25c753a764de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:14.416999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25595ba7-7741-4c96-935f-25c753a764de admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:14.435702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1238/2481] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:14 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:14.439874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:14.451189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a73e00b-6eb6-4297-b3cc-d1082ea9743f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:14.456386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:14.457054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.457333 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.457631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] PATH_INFO: `/v3/projects/6fbbb4538fc747fca4b3aa88962bcaa7/users/67586fbaaaec47f58dfdca3c4f841cfe/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.474774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fbbb4538fc747fca4b3aa88962bcaa7, user_id=67586fbaaaec47f58dfdca3c4f841cfe, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:14.477545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:14.481665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:14.503903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1244/2482] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:14 2022] PUT /identity/v3/projects/6fbbb4538fc747fca4b3aa88962bcaa7/users/67586fbaaaec47f58dfdca3c4f841cfe/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:14.515276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.515842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.515943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.546018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] MFA Rules not processed for user `67586fbaaaec47f58dfdca3c4f841cfe`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:14.547900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:14.622937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1239/2483] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:14 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:14.634486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:14.636543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:14.705301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c73157-0900-4845-809a-2a040623c0d4 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:14.708956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:14.709621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:14.709894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:14.710181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:14.719681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:14.721172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:14.730899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1245/2484] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:14 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:15.604127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:15.605762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:15.662514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:15.678229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05043f19-efb8-4f2e-a8b8-f186ec06d59a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:15.682641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:15.683305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:15.683576 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:15.683897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] PATH_INFO: `/v3/users/56c86f73f98a4fb2af787534fd47e2ed` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:15.698466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=56c86f73f98a4fb2af787534fd47e2ed)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:15.700455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:15.747232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56c86f73f98a4fb2af787534fd47e2ed'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:15.747900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56c86f73f98a4fb2af787534fd47e2ed'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:15.755772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56c86f73f98a4fb2af787534fd47e2ed'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:15.767408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56c86f73f98a4fb2af787534fd47e2ed'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:15.768051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c9d4e91f-79ca-4ad8-88fe-43e2add41e6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56c86f73f98a4fb2af787534fd47e2ed'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:15.780369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1240/2485] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:15 2022] DELETE /identity/v3/users/56c86f73f98a4fb2af787534fd47e2ed => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:15.960044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:15.976497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e2670-f60e-472e-8ac1-664a45c969a7 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.034389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.035041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.035316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.035584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] PATH_INFO: `/v3/projects/f53abe2cc7a94ddaadb68b26b57faf69` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.045802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f53abe2cc7a94ddaadb68b26b57faf69)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.048344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.094579 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f53abe2cc7a94ddaadb68b26b57faf69', 'request_id': 'req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:16.100177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f53abe2cc7a94ddaadb68b26b57faf69', 'request_id': 'req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:16.106828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f53abe2cc7a94ddaadb68b26b57faf69', 'request_id': 'req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:16.119850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] The token cache is being invalidate because project f53abe2cc7a94ddaadb68b26b57faf69 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:16.120129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:16.120753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-f003e9e6-52b9-4fa3-b5ad-8f9d2154bd58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:16.122245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1246/2486] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:15 2022] DELETE /identity/v3/projects/f53abe2cc7a94ddaadb68b26b57faf69 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:16.127294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7324971b-5f1d-45e2-80cc-8b45858e3e33 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.127582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7324971b-5f1d-45e2-80cc-8b45858e3e33 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.127860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7324971b-5f1d-45e2-80cc-8b45858e3e33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.147315 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7324971b-5f1d-45e2-80cc-8b45858e3e33 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:16.148503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7324971b-5f1d-45e2-80cc-8b45858e3e33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:16.215595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1241/2487] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:16.221456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc388f-d4a6-4aef-8b77-80a8a92c9d2a None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.221900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc388f-d4a6-4aef-8b77-80a8a92c9d2a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.222175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc388f-d4a6-4aef-8b77-80a8a92c9d2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.242926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-bafc388f-d4a6-4aef-8b77-80a8a92c9d2a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:16.244630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-bafc388f-d4a6-4aef-8b77-80a8a92c9d2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:16.254324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1247/2488] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:16.258628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7324971b-5f1d-45e2-80cc-8b45858e3e33 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.269406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7324971b-5f1d-45e2-80cc-8b45858e3e33 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.274278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.274871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.275184 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.275468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.276095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:16.276425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.278021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.290475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1242/2489] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:43:16.294674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafc388f-d4a6-4aef-8b77-80a8a92c9d2a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.304035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafc388f-d4a6-4aef-8b77-80a8a92c9d2a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.308633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.309222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.309496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.309761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.310396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.312399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.323513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:16.342209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1248/2490] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:16 2022] POST /identity/v3/projects => generated 363 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:43:16.347068 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.357146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23de75c-8dbe-4a27-a087-9a1a6d068f8e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.403813 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:16.419747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.420352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.420625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.420893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.421515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.423894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.446378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1243/2491] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:16 2022] POST /identity/v3/users => generated 282 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:16.450617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.461771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c4267-d2ec-471d-a8e7-022b4658dd58 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.468201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.469401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.469998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.470132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.470700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.472312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.482590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1249/2492] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:16.488195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.503119 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403802ed-ea39-4010-9621-bcdf43087c31 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.508253 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb850509-3e15-48c2-9d68-71167a0c4192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.508544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb850509-3e15-48c2-9d68-71167a0c4192 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.508822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb850509-3e15-48c2-9d68-71167a0c4192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.509088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb850509-3e15-48c2-9d68-71167a0c4192 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.509755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb850509-3e15-48c2-9d68-71167a0c4192 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.511370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb850509-3e15-48c2-9d68-71167a0c4192 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.525309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1244/2493] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:16.531399 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.541860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebb4e16-472d-42ef-8104-127fbf4378bb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.547785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.548522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.548862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.549200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] PATH_INFO: `/v3/projects/adf41d5c26ae4c3fb20f700ce793de18/users/34465e19e9c043a0a7ca932def563a95/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.570549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=adf41d5c26ae4c3fb20f700ce793de18, user_id=34465e19e9c043a0a7ca932def563a95, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.573534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.587595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1250/2494] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:16 2022] PUT /identity/v3/projects/adf41d5c26ae4c3fb20f700ce793de18/users/34465e19e9c043a0a7ca932def563a95/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:16.599384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed78e9-30fb-4e16-8ef5-6591d85ab41d None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.599816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed78e9-30fb-4e16-8ef5-6591d85ab41d None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.600161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed78e9-30fb-4e16-8ef5-6591d85ab41d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.629259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-faed78e9-30fb-4e16-8ef5-6591d85ab41d None None] MFA Rules not processed for user `34465e19e9c043a0a7ca932def563a95`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:16.630472 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-faed78e9-30fb-4e16-8ef5-6591d85ab41d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:16.703139 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1245/2495] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:16 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:16.705859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.770795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c63a205-76c9-4d68-a908-bfa1448ce7a8 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.775919 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.776520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.776794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.777065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.777690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.779625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.806244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1251/2496] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:16 2022] POST /identity/v3/projects => generated 365 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:16.810350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed78e9-30fb-4e16-8ef5-6591d85ab41d None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.821780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed78e9-30fb-4e16-8ef5-6591d85ab41d None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.877616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.878207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.878488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.878759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.879419 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.881325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.901695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1246/2497] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:16 2022] POST /identity/v3/users => generated 289 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:16.905733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.915960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839b8699-97f0-4b95-b7fa-b0f6ed568552 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.920609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.921202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.921484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.921759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.923145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.924740 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.935330 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1252/2498] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:16.939947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:16.949930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf585a1a-2b73-42a4-96ec-730020b951fc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:16.954552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:16.955239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:16.955515 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:16.955784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] PATH_INFO: `/v3/projects/9a5323298ee34daab53434d68948f6db/users/ec62bcbd9d324146ad34dca56216a8c1/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:16.970467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a5323298ee34daab53434d68948f6db, user_id=ec62bcbd9d324146ad34dca56216a8c1, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:16.973127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:16.987027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1247/2499] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:16 2022] PUT /identity/v3/projects/9a5323298ee34daab53434d68948f6db/users/ec62bcbd9d324146ad34dca56216a8c1/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:16.990870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:17.001721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a886cf-4c6f-43d0-aced-03b4630b20a0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:17.056418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adcf02a-8110-4168-973d-976fff2c3e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:17.057017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adcf02a-8110-4168-973d-976fff2c3e3a admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:17.057293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adcf02a-8110-4168-973d-976fff2c3e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:17.057561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adcf02a-8110-4168-973d-976fff2c3e3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:17.058190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adcf02a-8110-4168-973d-976fff2c3e3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:17.059812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adcf02a-8110-4168-973d-976fff2c3e3a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:17.069997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1253/2500] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:16 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:17.074154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:17.084281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255ac9b1-fe20-4652-8ab5-bcb871be1d33 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:17.088823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:17.089451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:17.089725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:17.089995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] PATH_INFO: `/v3/projects/9a5323298ee34daab53434d68948f6db/users/ec62bcbd9d324146ad34dca56216a8c1/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:17.091891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a5323298ee34daab53434d68948f6db, user_id=ec62bcbd9d324146ad34dca56216a8c1, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:17.094528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:17.104975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1248/2501] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:17 2022] PUT /identity/v3/projects/9a5323298ee34daab53434d68948f6db/users/ec62bcbd9d324146ad34dca56216a8c1/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:17.117359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed947e33-d5ab-451e-a3a3-6a500893f5c3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:17.117652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed947e33-d5ab-451e-a3a3-6a500893f5c3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:17.117927 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed947e33-d5ab-451e-a3a3-6a500893f5c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:17.146231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ed947e33-d5ab-451e-a3a3-6a500893f5c3 None None] MFA Rules not processed for user `ec62bcbd9d324146ad34dca56216a8c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:17.147488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ed947e33-d5ab-451e-a3a3-6a500893f5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:17.160147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:17.216906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1254/2502] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:17 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:17.228894 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:17.230687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:17.241206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:17.290394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7aae78-a960-4e1e-88c1-48105e4cf776 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:17.294577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65243ca-d6cf-436e-b816-34985b1d8631 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:17.295207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65243ca-d6cf-436e-b816-34985b1d8631 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:17.295479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65243ca-d6cf-436e-b816-34985b1d8631 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:17.295771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65243ca-d6cf-436e-b816-34985b1d8631 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:17.305630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65243ca-d6cf-436e-b816-34985b1d8631 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:17.307069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65243ca-d6cf-436e-b816-34985b1d8631 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:17.316904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1249/2503] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:17 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:17.748314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed947e33-d5ab-451e-a3a3-6a500893f5c3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:17.817026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed947e33-d5ab-451e-a3a3-6a500893f5c3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:17.822639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:17.823096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:17.823380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:17.823652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] PATH_INFO: `/v3/users/34465e19e9c043a0a7ca932def563a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:17.824840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34465e19e9c043a0a7ca932def563a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:17.826933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:17.876770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34465e19e9c043a0a7ca932def563a95'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:17.877490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34465e19e9c043a0a7ca932def563a95'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:17.885455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34465e19e9c043a0a7ca932def563a95'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:17.886240 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34465e19e9c043a0a7ca932def563a95'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:17.899331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34465e19e9c043a0a7ca932def563a95'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:17.915581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1255/2504] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:17 2022] DELETE /identity/v3/users/34465e19e9c043a0a7ca932def563a95 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:17.917468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:17.917916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:17.918127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:17.941661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:17.942947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:18.008025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1250/2505] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:18.020218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:18.022328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:18.057715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:18.084266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1058fa9b-33d9-413c-a6b7-bd4f487cb500 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:18.088158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:18.088765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:18.089052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:18.089319 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:18.099098 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:18.100536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:18.109974 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1256/2506] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:18.390239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:18.392369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:18.404220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e64d55-c1ff-4257-8bf2-90dbe5de8985 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:18.410712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:18.411511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:18.411882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:18.412221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] PATH_INFO: `/v3/projects/adf41d5c26ae4c3fb20f700ce793de18` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:18.413503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=adf41d5c26ae4c3fb20f700ce793de18)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:18.415468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:18.463398 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adf41d5c26ae4c3fb20f700ce793de18', 'request_id': 'req-3965b05b-e997-4957-b26f-741ed70f49b4'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.469836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adf41d5c26ae4c3fb20f700ce793de18', 'request_id': 'req-3965b05b-e997-4957-b26f-741ed70f49b4'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.475458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adf41d5c26ae4c3fb20f700ce793de18', 'request_id': 'req-3965b05b-e997-4957-b26f-741ed70f49b4'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.488155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] The token cache is being invalidate because project adf41d5c26ae4c3fb20f700ce793de18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:18.488440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.489050 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.490529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1251/2507] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:18 2022] DELETE /identity/v3/projects/adf41d5c26ae4c3fb20f700ce793de18 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:18.494878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:18.507834 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5941fa-0006-4e8e-b224-e8fcf5c68d9f service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:18.563077 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:18.564038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:18.564546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:18.564995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] PATH_INFO: `/v3/users/ec62bcbd9d324146ad34dca56216a8c1` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:18.566129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec62bcbd9d324146ad34dca56216a8c1)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:18.568126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:18.614532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec62bcbd9d324146ad34dca56216a8c1'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.615199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec62bcbd9d324146ad34dca56216a8c1'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.622723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec62bcbd9d324146ad34dca56216a8c1'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.623392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec62bcbd9d324146ad34dca56216a8c1'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.637111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5fbec672-5a96-4b02-b92e-9d3421e5c36b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec62bcbd9d324146ad34dca56216a8c1'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:18.649167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1257/2508] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:18 2022] DELETE /identity/v3/users/ec62bcbd9d324146ad34dca56216a8c1 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:43:18.927156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:18.928671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:18.942848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3965b05b-e997-4957-b26f-741ed70f49b4 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:18.973245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:19.004036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.004656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.004931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.005202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] PATH_INFO: `/v3/projects/9a5323298ee34daab53434d68948f6db` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.006307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a5323298ee34daab53434d68948f6db)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.008249 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.061586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a5323298ee34daab53434d68948f6db', 'request_id': 'req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:19.068223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a5323298ee34daab53434d68948f6db', 'request_id': 'req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:19.074251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a5323298ee34daab53434d68948f6db', 'request_id': 'req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:19.089873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] The token cache is being invalidate because project 9a5323298ee34daab53434d68948f6db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:19.090299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:19.091144 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ca1ff196-a19c-4ce3-8f7e-f173b63bc6e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:19.092727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1252/2509] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:18 2022] DELETE /identity/v3/projects/9a5323298ee34daab53434d68948f6db => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:19.098141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec186bc-37e3-439a-893d-c89e811d9dee None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.098677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec186bc-37e3-439a-893d-c89e811d9dee None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.098774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec186bc-37e3-439a-893d-c89e811d9dee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.123337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-aec186bc-37e3-439a-893d-c89e811d9dee None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:19.124404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-aec186bc-37e3-439a-893d-c89e811d9dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:19.176255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:19.190464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1258/2510] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:43:19.198136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a391190b-9aea-429d-b6f6-75e2c2864b2a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.198429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a391190b-9aea-429d-b6f6-75e2c2864b2a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.198705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a391190b-9aea-429d-b6f6-75e2c2864b2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.246009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a391190b-9aea-429d-b6f6-75e2c2864b2a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:19.247507 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a391190b-9aea-429d-b6f6-75e2c2864b2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:19.255839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1253/2511] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:19.260209 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec186bc-37e3-439a-893d-c89e811d9dee None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.270764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec186bc-37e3-439a-893d-c89e811d9dee None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.275396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.275990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.276266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.276536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.277173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:19.277473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.279315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.291776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1259/2512] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:19.296109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a391190b-9aea-429d-b6f6-75e2c2864b2a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.306191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a391190b-9aea-429d-b6f6-75e2c2864b2a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.310845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.311462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.311736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.312004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.312691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.314773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.342989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1254/2513] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:19 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:19.347936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.357783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346bcbe-b407-4944-a051-4cc0a61c3233 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.419048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.419669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.419981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.420255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.420887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.423409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.448952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1260/2514] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:19 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:19.450011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.461237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a1e80a-d44b-4441-8afd-6a4e172fdde5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.465958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.466541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.466819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.467117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.467749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.470159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.480041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1255/2515] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:43:19.484076 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.494700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca351947-047d-42c3-8979-d2b2eb87e4bd admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.499350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837e23d-145b-4acd-94ff-5257432c2d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.499937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837e23d-145b-4acd-94ff-5257432c2d5d admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.500211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837e23d-145b-4acd-94ff-5257432c2d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.500616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837e23d-145b-4acd-94ff-5257432c2d5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.501296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837e23d-145b-4acd-94ff-5257432c2d5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.502929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837e23d-145b-4acd-94ff-5257432c2d5d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.514996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1261/2516] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:19.519096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.529687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de574f-58cb-4f09-a46c-a82b33a86e7e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.535819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.536466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.536743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.537020 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] PATH_INFO: `/v3/projects/af4aec1e8cf74fc3bcfefc5e2aec9742/users/0dbbc9e674f0451c996c9e025f187cc2/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.552682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] RBAC: Authorizing `identity:create_grant(project_id=af4aec1e8cf74fc3bcfefc5e2aec9742, user_id=0dbbc9e674f0451c996c9e025f187cc2, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.555305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.570578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1256/2517] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:19 2022] PUT /identity/v3/projects/af4aec1e8cf74fc3bcfefc5e2aec9742/users/0dbbc9e674f0451c996c9e025f187cc2/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:19.581550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ab6bfc-fb40-4358-94a0-4c812dbaf81f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.581823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ab6bfc-fb40-4358-94a0-4c812dbaf81f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.582104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ab6bfc-fb40-4358-94a0-4c812dbaf81f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.612018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a7ab6bfc-fb40-4358-94a0-4c812dbaf81f None None] MFA Rules not processed for user `0dbbc9e674f0451c996c9e025f187cc2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:19.613123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a7ab6bfc-fb40-4358-94a0-4c812dbaf81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:19.680562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1262/2518] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:19.685099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.751230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151aa444-4216-4b58-bd97-980514d0e15c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.755959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.756582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.756861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.757132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.757757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.759678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.764222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:19.789713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1257/2519] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:19 2022] POST /identity/v3/projects => generated 387 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:19.793841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ab6bfc-fb40-4358-94a0-4c812dbaf81f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.805355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ab6bfc-fb40-4358-94a0-4c812dbaf81f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.854544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:19.867526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.868135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.868417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.868688 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.869575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.871533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.894619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1263/2520] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:19 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:19.898796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.910324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0dfcb-303a-4c62-86b1-ca5e5eb2d8cd admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.915890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.916480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.916753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.917019 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.917649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.919317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.930775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1258/2521] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:19.934216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.945413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56bc5b-44a4-4316-bff1-a532c6ead94e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.949975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50b7724-b812-4f1e-a5a8-6a883d83eed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.950563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b7724-b812-4f1e-a5a8-6a883d83eed8 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.950836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b7724-b812-4f1e-a5a8-6a883d83eed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.951140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50b7724-b812-4f1e-a5a8-6a883d83eed8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:19.951773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50b7724-b812-4f1e-a5a8-6a883d83eed8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:19.953412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50b7724-b812-4f1e-a5a8-6a883d83eed8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:19.963537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1264/2522] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:19.967631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:19.977923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffd423a-11f2-4db0-a17f-c178753b2dbf admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:19.982538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:19.983298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:19.983604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:19.983882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] PATH_INFO: `/v3/projects/ba8f1c7957ed49fe9ff50c5db39c2d74/users/104857221c914ab6a14647f2ad3fa7fe/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.001291 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba8f1c7957ed49fe9ff50c5db39c2d74, user_id=104857221c914ab6a14647f2ad3fa7fe, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.004202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.020114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1259/2523] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:19 2022] PUT /identity/v3/projects/ba8f1c7957ed49fe9ff50c5db39c2d74/users/104857221c914ab6a14647f2ad3fa7fe/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:20.032196 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c4258-39a2-4dda-83c4-e0fa1f8ecdc1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.032817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c4258-39a2-4dda-83c4-e0fa1f8ecdc1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.033140 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56c4258-39a2-4dda-83c4-e0fa1f8ecdc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.064117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a56c4258-39a2-4dda-83c4-e0fa1f8ecdc1 None None] MFA Rules not processed for user `104857221c914ab6a14647f2ad3fa7fe`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:20.065237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a56c4258-39a2-4dda-83c4-e0fa1f8ecdc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:20.127637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1265/2524] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:20 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:20.132100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:20.193054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2640cf-0d9f-4373-ba29-e7470757af3c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:20.197782 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:20.198369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.198647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.198920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.199597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.201497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.231538 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1260/2525] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:20 2022] POST /identity/v3/projects => generated 385 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:20.232600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56c4258-39a2-4dda-83c4-e0fa1f8ecdc1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:20.245137 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56c4258-39a2-4dda-83c4-e0fa1f8ecdc1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:20.303714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:20.304317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.304612 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.304878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.305513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.307603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.329628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1266/2526] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:20 2022] POST /identity/v3/users => generated 299 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:20.333833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:20.344573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a21342-3f3e-4592-b6c1-86f1b1814263 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:20.349176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:20.349756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.350032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.350299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.350924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.352529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.363074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1261/2527] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:20.367194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:20.378131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a73b4e2-a933-401f-b10c-ff45c3a274d0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:20.382663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:20.383350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.383625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.383898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] PATH_INFO: `/v3/projects/33a9a57ff58e42059e85bc0453e1b46a/users/5e2b4aa907e54621ae93df92d37f1f01/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.399074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33a9a57ff58e42059e85bc0453e1b46a, user_id=5e2b4aa907e54621ae93df92d37f1f01, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.401728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.413980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1267/2528] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:20 2022] PUT /identity/v3/projects/33a9a57ff58e42059e85bc0453e1b46a/users/5e2b4aa907e54621ae93df92d37f1f01/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:20.417880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:20.429552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae4d6b-c74a-4777-97fb-23b09e2b201e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:20.489762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f43297-70a9-47de-a549-84bafee54d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:20.490362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f43297-70a9-47de-a549-84bafee54d3b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.490638 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f43297-70a9-47de-a549-84bafee54d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.490911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f43297-70a9-47de-a549-84bafee54d3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.491704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f43297-70a9-47de-a549-84bafee54d3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.493281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f43297-70a9-47de-a549-84bafee54d3b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.504023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1262/2529] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:20 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:20.508222 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:20.522127 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91157a9f-3bb0-4abc-bcaa-2a92b1225315 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:20.527414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:20.528061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.528342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.528610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] PATH_INFO: `/v3/projects/33a9a57ff58e42059e85bc0453e1b46a/users/5e2b4aa907e54621ae93df92d37f1f01/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.530494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33a9a57ff58e42059e85bc0453e1b46a, user_id=5e2b4aa907e54621ae93df92d37f1f01, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.533362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.543580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1268/2530] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:20 2022] PUT /identity/v3/projects/33a9a57ff58e42059e85bc0453e1b46a/users/5e2b4aa907e54621ae93df92d37f1f01/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:20.553598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.553954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.554292 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.582606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] MFA Rules not processed for user `5e2b4aa907e54621ae93df92d37f1f01`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:20.583830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:20.592120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:20.653630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1263/2531] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:20 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:20.664850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:20.666564 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:20.677852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:20.732337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6203396-ec28-4896-b3b4-d69dc91a61c0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:20.736729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:20.737339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:20.737605 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:20.737872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:20.747213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:20.748632 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:20.758615 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1269/2532] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:20 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:21.278066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:21.279475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:21.291801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410fdf2d-883b-4b96-a785-0df2281cc3dd None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:21.295501 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4c7591-3b12-4e62-ae0f-62b950cd6934 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:21.296095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4c7591-3b12-4e62-ae0f-62b950cd6934 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:21.296370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4c7591-3b12-4e62-ae0f-62b950cd6934 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:21.296702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4c7591-3b12-4e62-ae0f-62b950cd6934 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:21.353902 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4c7591-3b12-4e62-ae0f-62b950cd6934 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:21.355290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4c7591-3b12-4e62-ae0f-62b950cd6934 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:21.365033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1264/2533] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:23.366995 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:23.368431 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:23.431581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb89d9d8-c3f5-4996-b68a-9c55d61e9f24 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:23.435991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:23.436601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:23.436880 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:23.437155 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] PATH_INFO: `/v3/users/67586fbaaaec47f58dfdca3c4f841cfe` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:23.438759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=67586fbaaaec47f58dfdca3c4f841cfe)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:23.440696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:23.487630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '67586fbaaaec47f58dfdca3c4f841cfe'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:23.488187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '67586fbaaaec47f58dfdca3c4f841cfe'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:23.495361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '67586fbaaaec47f58dfdca3c4f841cfe'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:23.495998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '67586fbaaaec47f58dfdca3c4f841cfe'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:23.508260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '67586fbaaaec47f58dfdca3c4f841cfe'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:23.521211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1270/2534] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:23 2022] DELETE /identity/v3/users/67586fbaaaec47f58dfdca3c4f841cfe => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:23.525687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:23.526152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:23.526270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:23.544958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:23.546192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:23.603321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1265/2535] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:23.613126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:23.614563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:23.650009 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:23.675034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0834edb6-86d5-4da9-9184-4444b293da65 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:23.678644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4646ddfc-d776-461c-9d4e-87f31ceb7e68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:23.679285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646ddfc-d776-461c-9d4e-87f31ceb7e68 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:23.679562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646ddfc-d776-461c-9d4e-87f31ceb7e68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:23.679832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4646ddfc-d776-461c-9d4e-87f31ceb7e68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:23.688697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4646ddfc-d776-461c-9d4e-87f31ceb7e68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:23.690111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4646ddfc-d776-461c-9d4e-87f31ceb7e68 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:23.699329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1271/2536] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:23.976945 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:23.979188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:23.992451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4e2136-537c-483b-8f5a-cd2d9d53392f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:23.997829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:23.998512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:23.998796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:23.999098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] PATH_INFO: `/v3/projects/6fbbb4538fc747fca4b3aa88962bcaa7` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.000422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6fbbb4538fc747fca4b3aa88962bcaa7)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.002387 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.049853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:24.059233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6fbbb4538fc747fca4b3aa88962bcaa7', 'request_id': 'req-bb9153d1-c93b-474a-8164-44e4524f6656'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:24.066608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6fbbb4538fc747fca4b3aa88962bcaa7', 'request_id': 'req-bb9153d1-c93b-474a-8164-44e4524f6656'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:24.072736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6fbbb4538fc747fca4b3aa88962bcaa7', 'request_id': 'req-bb9153d1-c93b-474a-8164-44e4524f6656'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:24.085691 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] The token cache is being invalidate because project 6fbbb4538fc747fca4b3aa88962bcaa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:24.085980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:24.086598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bb9153d1-c93b-474a-8164-44e4524f6656 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:24.088157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1266/2537] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:23 2022] DELETE /identity/v3/projects/6fbbb4538fc747fca4b3aa88962bcaa7 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:24.093259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c228041-bed9-4ea1-a023-77da9a624d57 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.094139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c228041-bed9-4ea1-a023-77da9a624d57 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.094484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c228041-bed9-4ea1-a023-77da9a624d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.118062 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5c228041-bed9-4ea1-a023-77da9a624d57 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:24.119552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5c228041-bed9-4ea1-a023-77da9a624d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:24.185603 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1272/2538] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:24.193817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2095fdb-026e-409e-8816-52ac94537bea None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.194102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2095fdb-026e-409e-8816-52ac94537bea None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.194423 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2095fdb-026e-409e-8816-52ac94537bea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.216510 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-a2095fdb-026e-409e-8816-52ac94537bea None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:24.217719 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a2095fdb-026e-409e-8816-52ac94537bea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:24.226625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1267/2539] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:24.231026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c228041-bed9-4ea1-a023-77da9a624d57 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.241502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c228041-bed9-4ea1-a023-77da9a624d57 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.246656 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.247232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.247506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.247774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.248418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:24.248718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.250878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.262600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1273/2540] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:24.267209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2095fdb-026e-409e-8816-52ac94537bea None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.276764 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2095fdb-026e-409e-8816-52ac94537bea None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.281909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.282656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.282930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.283239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.283863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.285747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.311918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1268/2541] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:24 2022] POST /identity/v3/projects => generated 371 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:24.316243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.326247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db613009-6f0e-42d9-a2c2-58ff21b64f5b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.382494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.383165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.383447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.383698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.384339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.386262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.407168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1274/2542] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:24 2022] POST /identity/v3/users => generated 286 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:24.411426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.422250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfd26c4-facf-4786-8b10-d27b6a7b3f3d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.426785 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.427402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.427679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.427962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.428578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.430155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.439910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1269/2543] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:24.443681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.454069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887ae241-4d2b-4cdf-bd76-0def735b2a8a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.458576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae2fdaf-669f-4856-b67c-009fd3732ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.459198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2fdaf-669f-4856-b67c-009fd3732ee2 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.459473 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2fdaf-669f-4856-b67c-009fd3732ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.459744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae2fdaf-669f-4856-b67c-009fd3732ee2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.460372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae2fdaf-669f-4856-b67c-009fd3732ee2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.461952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae2fdaf-669f-4856-b67c-009fd3732ee2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.473064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1275/2544] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:24.477064 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.487221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a617eb1-efc8-40d9-a1fc-356075ab41e1 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.491766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.492392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.492668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.492938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] PATH_INFO: `/v3/projects/5626a34eb04e460b89f3d4a01d9d1c80/users/8734279f63be407c892cbd8b9ec149c1/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.508008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5626a34eb04e460b89f3d4a01d9d1c80, user_id=8734279f63be407c892cbd8b9ec149c1, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.510629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.524044 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1270/2545] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:24 2022] PUT /identity/v3/projects/5626a34eb04e460b89f3d4a01d9d1c80/users/8734279f63be407c892cbd8b9ec149c1/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:24.536381 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b4e0e5-e781-44a2-9354-2cabc0ff65f5 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.536664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b4e0e5-e781-44a2-9354-2cabc0ff65f5 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.536964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b4e0e5-e781-44a2-9354-2cabc0ff65f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.566548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b4b4e0e5-e781-44a2-9354-2cabc0ff65f5 None None] MFA Rules not processed for user `8734279f63be407c892cbd8b9ec149c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:24.567771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b4b4e0e5-e781-44a2-9354-2cabc0ff65f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:24.596048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:24.636952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1276/2546] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:24 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:24.642202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.705762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae0736-b0f9-4aa2-aa5e-c710b60fa183 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.710463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.711086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.711362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.711632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.712247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.714163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.740595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1271/2547] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:24 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:24.744871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b4e0e5-e781-44a2-9354-2cabc0ff65f5 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.756939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b4e0e5-e781-44a2-9354-2cabc0ff65f5 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.811129 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.811661 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.811954 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.812790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.813402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.815336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.835736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1277/2548] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:24 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:24.839816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.851178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41267183-80b7-4f95-a904-8d0a09f653e4 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.855818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.856403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.856678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.856949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.857588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.859822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.870582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1272/2549] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:24.874858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.884649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367b3d9-2890-45bf-be6f-75c92b2887ae admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.889511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.890152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.890430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.890696 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] PATH_INFO: `/v3/projects/cc87f5aac6254615a4b3bc27303b8407/users/502bcb154c28416c9e1523fee0739f38/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.905515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc87f5aac6254615a4b3bc27303b8407, user_id=502bcb154c28416c9e1523fee0739f38, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.908858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:24.921083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1278/2550] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:24 2022] PUT /identity/v3/projects/cc87f5aac6254615a4b3bc27303b8407/users/502bcb154c28416c9e1523fee0739f38/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:24.925193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:24.934900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e51b53-6e29-4d55-96a2-96f3e454bf81 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:24.968466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:24.993152 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ea24b6-29ad-4ab6-b85e-eae2a1809839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:24.993739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea24b6-29ad-4ab6-b85e-eae2a1809839 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:24.994012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea24b6-29ad-4ab6-b85e-eae2a1809839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:24.994525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ea24b6-29ad-4ab6-b85e-eae2a1809839 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:24.995189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ea24b6-29ad-4ab6-b85e-eae2a1809839 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:24.996790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ea24b6-29ad-4ab6-b85e-eae2a1809839 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:25.007215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1273/2551] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:24 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:25.011546 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:25.024245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fca750-edfd-47e1-b828-b4503a0fc396 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:25.028849 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:25.029487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:25.029764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:25.030037 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] PATH_INFO: `/v3/projects/cc87f5aac6254615a4b3bc27303b8407/users/502bcb154c28416c9e1523fee0739f38/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:25.032122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc87f5aac6254615a4b3bc27303b8407, user_id=502bcb154c28416c9e1523fee0739f38, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:25.034741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:25.044816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1279/2552] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:25 2022] PUT /identity/v3/projects/cc87f5aac6254615a4b3bc27303b8407/users/502bcb154c28416c9e1523fee0739f38/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:25.056591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d719925-08d9-4156-bf59-67103261a7fa None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:25.056893 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d719925-08d9-4156-bf59-67103261a7fa None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:25.057168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d719925-08d9-4156-bf59-67103261a7fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:25.086043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6d719925-08d9-4156-bf59-67103261a7fa None None] MFA Rules not processed for user `502bcb154c28416c9e1523fee0739f38`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:25.087310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6d719925-08d9-4156-bf59-67103261a7fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:25.154488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1274/2553] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:25 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:25.166138 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:25.167858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:25.227391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e32f5-16bf-423a-976e-9ed74ad1344e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:25.231325 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:25.231933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:25.232208 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:25.232482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:25.242569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:25.244014 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:25.254952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1280/2554] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:25 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:25.331212 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d719925-08d9-4156-bf59-67103261a7fa None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:25.340973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d719925-08d9-4156-bf59-67103261a7fa None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:25.344917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:25.345512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:25.345791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:25.346180 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:25.411533 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:25.412980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:25.423123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1275/2555] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:25 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:25.456504 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:25.457937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:25.471538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a91c11-0461-474e-941b-d647592a3bfb service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:25.476131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eca99d-830e-45bb-8074-d33614c1ce60 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:25.476846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eca99d-830e-45bb-8074-d33614c1ce60 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:25.477170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eca99d-830e-45bb-8074-d33614c1ce60 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:25.477547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eca99d-830e-45bb-8074-d33614c1ce60 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:25.479906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e0eca99d-830e-45bb-8074-d33614c1ce60 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:25.507792 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:25.570796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eca99d-830e-45bb-8074-d33614c1ce60 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:25.572360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eca99d-830e-45bb-8074-d33614c1ce60 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:25.581835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1281/2556] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:25 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:27.649713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:27.651199 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:27.713132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:27.717639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3512ae1-4e56-43c0-aead-3f3f75270cea service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:27.723458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:27.724125 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:27.724448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:27.724760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] PATH_INFO: `/v3/users/0dbbc9e674f0451c996c9e025f187cc2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:27.726017 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0dbbc9e674f0451c996c9e025f187cc2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:27.728021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:27.772639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0dbbc9e674f0451c996c9e025f187cc2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:27.773274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0dbbc9e674f0451c996c9e025f187cc2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:27.780201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0dbbc9e674f0451c996c9e025f187cc2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:27.791807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0dbbc9e674f0451c996c9e025f187cc2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:27.792424 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0dbbc9e674f0451c996c9e025f187cc2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:27.805372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1276/2557] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:27 2022] DELETE /identity/v3/users/0dbbc9e674f0451c996c9e025f187cc2 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:27.809889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:27.810372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:27.810503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:27.829020 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:27.830201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:27.887444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1282/2558] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:27.897143 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:27.898522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:27.954607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc60501-f2e3-4c63-a107-66e1bdf7a481 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:27.958216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:27.958802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:27.959115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:27.959385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:27.970004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:27.971414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:27.981525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1277/2559] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:28.263932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:28.265422 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:28.280342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b06f5af-632d-4bfc-9ca8-fb3e1e791b0d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:28.285541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:28.286174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:28.286447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:28.286714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] PATH_INFO: `/v3/projects/af4aec1e8cf74fc3bcfefc5e2aec9742` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:28.287865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] RBAC: Authorizing `identity:delete_project(project_id=af4aec1e8cf74fc3bcfefc5e2aec9742)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:28.289755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:28.335893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af4aec1e8cf74fc3bcfefc5e2aec9742', 'request_id': 'req-e230166a-5df0-4e1a-92c8-85ab24f162db'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.341328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af4aec1e8cf74fc3bcfefc5e2aec9742', 'request_id': 'req-e230166a-5df0-4e1a-92c8-85ab24f162db'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.347692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af4aec1e8cf74fc3bcfefc5e2aec9742', 'request_id': 'req-e230166a-5df0-4e1a-92c8-85ab24f162db'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.360275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] The token cache is being invalidate because project af4aec1e8cf74fc3bcfefc5e2aec9742 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:28.360571 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.361186 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.362715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1283/2560] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:28 2022] DELETE /identity/v3/projects/af4aec1e8cf74fc3bcfefc5e2aec9742 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:28.366648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:28.377168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d731f02e-d354-4542-ae43-85a4cae3e30e service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:28.432411 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:28.433030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:28.433307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:28.433578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] PATH_INFO: `/v3/users/104857221c914ab6a14647f2ad3fa7fe` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:28.434707 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=104857221c914ab6a14647f2ad3fa7fe)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:28.436685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:28.458470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:28.483161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '104857221c914ab6a14647f2ad3fa7fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.483736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '104857221c914ab6a14647f2ad3fa7fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.491111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '104857221c914ab6a14647f2ad3fa7fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.503535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '104857221c914ab6a14647f2ad3fa7fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.504158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '104857221c914ab6a14647f2ad3fa7fe'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.516238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1278/2561] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:28 2022] DELETE /identity/v3/users/104857221c914ab6a14647f2ad3fa7fe => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:28.796103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:28.799309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:28.818141 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e230166a-5df0-4e1a-92c8-85ab24f162db admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:28.865478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:28.879648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:28.880266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:28.880539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:28.880807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] PATH_INFO: `/v3/projects/ba8f1c7957ed49fe9ff50c5db39c2d74` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:28.881905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba8f1c7957ed49fe9ff50c5db39c2d74)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:28.883838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:28.929414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba8f1c7957ed49fe9ff50c5db39c2d74', 'request_id': 'req-3856d897-97ff-4201-b150-641d0489a39a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.934833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba8f1c7957ed49fe9ff50c5db39c2d74', 'request_id': 'req-3856d897-97ff-4201-b150-641d0489a39a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.941094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba8f1c7957ed49fe9ff50c5db39c2d74', 'request_id': 'req-3856d897-97ff-4201-b150-641d0489a39a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.954312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] The token cache is being invalidate because project ba8f1c7957ed49fe9ff50c5db39c2d74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:28.954599 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.955256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:28.956733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1284/2562] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:28 2022] DELETE /identity/v3/projects/ba8f1c7957ed49fe9ff50c5db39c2d74 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:43:28.960871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:28.962285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:28.975823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79094c-c3d2-4a7d-aa3f-9ee7e0a8efb0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.036991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.037672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.037951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.038222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] PATH_INFO: `/v3/users/5e2b4aa907e54621ae93df92d37f1f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.039358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e2b4aa907e54621ae93df92d37f1f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.041254 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.087074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e2b4aa907e54621ae93df92d37f1f01'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.087979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e2b4aa907e54621ae93df92d37f1f01'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.095974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e2b4aa907e54621ae93df92d37f1f01'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.108233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e2b4aa907e54621ae93df92d37f1f01'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.108903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6d45d132-90e3-45bd-ba7e-b8569d9fbe3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e2b4aa907e54621ae93df92d37f1f01'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.121432 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1279/2563] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:28 2022] DELETE /identity/v3/users/5e2b4aa907e54621ae93df92d37f1f01 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:29.401607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:29.403211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:29.418244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3856d897-97ff-4201-b150-641d0489a39a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.475445 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.476067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.476340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.476609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] PATH_INFO: `/v3/projects/33a9a57ff58e42059e85bc0453e1b46a` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.477700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33a9a57ff58e42059e85bc0453e1b46a)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.479668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.524967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33a9a57ff58e42059e85bc0453e1b46a', 'request_id': 'req-15798df3-c786-432b-9fee-0ac202ec34f8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.530567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33a9a57ff58e42059e85bc0453e1b46a', 'request_id': 'req-15798df3-c786-432b-9fee-0ac202ec34f8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.537019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33a9a57ff58e42059e85bc0453e1b46a', 'request_id': 'req-15798df3-c786-432b-9fee-0ac202ec34f8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.549228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] The token cache is being invalidate because project 33a9a57ff58e42059e85bc0453e1b46a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:29.549569 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.550143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-15798df3-c786-432b-9fee-0ac202ec34f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:29.551691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1285/2564] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:29 2022] DELETE /identity/v3/projects/33a9a57ff58e42059e85bc0453e1b46a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:29.558386 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5338e3c-64d8-4a59-be24-eca4d733bc31 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.558669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5338e3c-64d8-4a59-be24-eca4d733bc31 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.558944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5338e3c-64d8-4a59-be24-eca4d733bc31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.579464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-c5338e3c-64d8-4a59-be24-eca4d733bc31 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:29.580676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c5338e3c-64d8-4a59-be24-eca4d733bc31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:29.592648 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:29.639973 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1280/2565] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:43:29.650044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8fdfd8-67e9-4d6b-a901-b6be5fc30120 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.650335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8fdfd8-67e9-4d6b-a901-b6be5fc30120 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.650623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8fdfd8-67e9-4d6b-a901-b6be5fc30120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.665184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:29.671165 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ee8fdfd8-67e9-4d6b-a901-b6be5fc30120 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:29.673645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ee8fdfd8-67e9-4d6b-a901-b6be5fc30120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:29.681604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1286/2566] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:29.686031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5338e3c-64d8-4a59-be24-eca4d733bc31 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:29.695839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5338e3c-64d8-4a59-be24-eca4d733bc31 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.700505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.701092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.701371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.701649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.702289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:29.702593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.704724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.716090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1281/2567] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:29.721801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8fdfd8-67e9-4d6b-a901-b6be5fc30120 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:29.731159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8fdfd8-67e9-4d6b-a901-b6be5fc30120 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.735908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.736499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.736771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.737038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.737687 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.739630 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.766403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1287/2568] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:29 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:29.770516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:29.780207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df2bedc-2fc4-49eb-93f6-248d2564f2d4 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.835703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.836290 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.836565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.836836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.837457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.839445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.860644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1282/2569] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:29 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:43:29.864721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:29.875302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79a6a45-2238-4a52-b851-483250d83587 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.879894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.880479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.880755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.881027 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.881655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.883274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.893482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1288/2570] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:29.897376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:29.907958 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a519f7ae-1725-4bd1-9070-4cc1f880c2c9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.912588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab677997-26a3-4d0c-ac4a-81f821a50016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.913168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab677997-26a3-4d0c-ac4a-81f821a50016 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.913444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab677997-26a3-4d0c-ac4a-81f821a50016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.913713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab677997-26a3-4d0c-ac4a-81f821a50016 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.914339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab677997-26a3-4d0c-ac4a-81f821a50016 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.915960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab677997-26a3-4d0c-ac4a-81f821a50016 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.926447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1283/2571] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:29.930540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:29.940376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415b292d-e584-4701-946a-3542a6fbc132 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:29.944894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:29.945526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.945809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.946069 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] PATH_INFO: `/v3/projects/d489a195eb55431ea8e591f485793061/users/4597766265db4e84b281401e100c47f9/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:29.960699 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d489a195eb55431ea8e591f485793061, user_id=4597766265db4e84b281401e100c47f9, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:29.963407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:29.976079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1289/2572] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:29 2022] PUT /identity/v3/projects/d489a195eb55431ea8e591f485793061/users/4597766265db4e84b281401e100c47f9/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:29.987650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ff37f-8ea7-4a51-a6c7-1664f7493efe None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:29.988130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ff37f-8ea7-4a51-a6c7-1664f7493efe None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:29.988230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ff37f-8ea7-4a51-a6c7-1664f7493efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.018269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-dc1ff37f-8ea7-4a51-a6c7-1664f7493efe None None] MFA Rules not processed for user `4597766265db4e84b281401e100c47f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:30.019659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-dc1ff37f-8ea7-4a51-a6c7-1664f7493efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:30.083282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1284/2573] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:30.087821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.148479 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5e3918-3a12-40ce-8462-a819da00bf44 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.153415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.154039 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.154317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.154582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.155232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.157258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.182862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1290/2574] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:30 2022] POST /identity/v3/projects => generated 377 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:30.186883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1ff37f-8ea7-4a51-a6c7-1664f7493efe None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.198211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1ff37f-8ea7-4a51-a6c7-1664f7493efe None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.207887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:30.256622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.257217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.257495 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.257767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.258394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.260364 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.287908 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1285/2575] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:30 2022] POST /identity/v3/users => generated 289 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:30.289320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.301116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc27d78-6190-4311-9f7b-bb5d0d1cc383 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.305962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.306559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.306836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.307142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.307816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.309419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.322056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1291/2576] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:30.326250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.337148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f774b18-b432-4572-97be-ba5ee06a98f1 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.342034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae5e6ee-bf6d-403d-8e61-4388a539614c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.342566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae5e6ee-bf6d-403d-8e61-4388a539614c admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.342845 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae5e6ee-bf6d-403d-8e61-4388a539614c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.343167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae5e6ee-bf6d-403d-8e61-4388a539614c admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.344371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae5e6ee-bf6d-403d-8e61-4388a539614c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.345950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae5e6ee-bf6d-403d-8e61-4388a539614c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.355822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1286/2577] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:30.360112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.370061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b3d41e-13dd-49c6-8639-8d9b5a31afd5 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.375717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.376363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.376644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.376921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] PATH_INFO: `/v3/projects/9f4ab9f71f434112895f78edfeeb6ac2/users/f4e1e94329b248b28f714120ef821532/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.391977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f4ab9f71f434112895f78edfeeb6ac2, user_id=f4e1e94329b248b28f714120ef821532, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.394673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.407351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1292/2578] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:30 2022] PUT /identity/v3/projects/9f4ab9f71f434112895f78edfeeb6ac2/users/f4e1e94329b248b28f714120ef821532/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:30.417703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba2b132-1f9b-4a19-90bd-96a905b3625e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.417990 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba2b132-1f9b-4a19-90bd-96a905b3625e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.418268 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba2b132-1f9b-4a19-90bd-96a905b3625e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.448061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0ba2b132-1f9b-4a19-90bd-96a905b3625e None None] MFA Rules not processed for user `f4e1e94329b248b28f714120ef821532`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:30.449597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0ba2b132-1f9b-4a19-90bd-96a905b3625e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:30.522786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1287/2579] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:30.527502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.584375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:30.596689 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2291c463-133e-49b8-9b33-2ef7a073c962 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.602440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.603164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.603510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.603850 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.604589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.606591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.637316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1293/2580] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:30 2022] POST /identity/v3/projects => generated 377 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:43:30.642288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba2b132-1f9b-4a19-90bd-96a905b3625e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.653406 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba2b132-1f9b-4a19-90bd-96a905b3625e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.713335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.713934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.714209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.714479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.715137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.717041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.738454 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1288/2581] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:30 2022] POST /identity/v3/users => generated 295 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:43:30.742446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.753646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccad192-7fdd-42bb-ae67-e73fe618469e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.758181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.758761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.759064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.759335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.759994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.761600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.772086 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1294/2582] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:30.776075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.786270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514bb260-e229-4375-b2cb-4e3eb8504431 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.790884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.791557 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.791835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.792111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] PATH_INFO: `/v3/projects/3173c879d2ae4d5abd8bbbdf4f1495a4/users/f9e3d11620f74b66a3f871079cb81e0f/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.800960 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:30.807883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3173c879d2ae4d5abd8bbbdf4f1495a4, user_id=f9e3d11620f74b66a3f871079cb81e0f, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.810982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.824338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1289/2583] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:30 2022] PUT /identity/v3/projects/3173c879d2ae4d5abd8bbbdf4f1495a4/users/f9e3d11620f74b66a3f871079cb81e0f/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:30.828193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.837961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7c7d28-599c-4448-b39d-1aebb354de9e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.893831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4697c59-6c6b-41a0-8c38-5e1ea8659954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.894446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4697c59-6c6b-41a0-8c38-5e1ea8659954 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.894618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4697c59-6c6b-41a0-8c38-5e1ea8659954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.894891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4697c59-6c6b-41a0-8c38-5e1ea8659954 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.895557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4697c59-6c6b-41a0-8c38-5e1ea8659954 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.897143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4697c59-6c6b-41a0-8c38-5e1ea8659954 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.907339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1295/2584] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:30 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:30.911791 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:30.922569 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c36dfb-ad0c-408c-ac61-cec784371f72 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:30.927184 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:30.927817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.928094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.928366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] PATH_INFO: `/v3/projects/3173c879d2ae4d5abd8bbbdf4f1495a4/users/f9e3d11620f74b66a3f871079cb81e0f/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.930260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] RBAC: Authorizing `identity:create_grant(project_id=3173c879d2ae4d5abd8bbbdf4f1495a4, user_id=f9e3d11620f74b66a3f871079cb81e0f, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:30.932909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:30.942982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1290/2585] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:30 2022] PUT /identity/v3/projects/3173c879d2ae4d5abd8bbbdf4f1495a4/users/f9e3d11620f74b66a3f871079cb81e0f/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:30.953797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:30.954145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:30.954486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:30.981932 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] MFA Rules not processed for user `f9e3d11620f74b66a3f871079cb81e0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:30.983148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:31.051809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1296/2586] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:30 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:31.063573 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:31.065317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:31.124331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7429f9f-d5f9-44da-a486-f04095e75bcd admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:31.128610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:31.129202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:31.129480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:31.129754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:31.138659 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:31.140197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:31.149714 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1291/2587] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:31 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:31.192469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:31.193833 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:31.206068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59da4e98-f7ef-4178-99ea-a640aa1e63ab None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:31.210007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:31.210678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:31.211003 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:31.211294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:31.258929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:31.274123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:31.275623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:31.285662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1297/2588] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:31.821394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:31.822832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:31.879211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c042017-3b28-44a2-a491-31b584010458 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:31.882795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5652a3ce-7c64-4814-bfe9-85cc7aefa040 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:31.883416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652a3ce-7c64-4814-bfe9-85cc7aefa040 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:31.883693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652a3ce-7c64-4814-bfe9-85cc7aefa040 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:31.883962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652a3ce-7c64-4814-bfe9-85cc7aefa040 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:31.885654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5652a3ce-7c64-4814-bfe9-85cc7aefa040 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:31.895789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5652a3ce-7c64-4814-bfe9-85cc7aefa040 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:31.897197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5652a3ce-7c64-4814-bfe9-85cc7aefa040 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:31.905034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1292/2589] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:35.401981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:35.474530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b64de-4c44-4f24-bcbe-9cd3f25d4a4d service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:35.483886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:35.484861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:35.485173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:35.485467 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] PATH_INFO: `/v3/users/4597766265db4e84b281401e100c47f9` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:35.487815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=4597766265db4e84b281401e100c47f9)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:35.489816 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:35.547864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4597766265db4e84b281401e100c47f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:35.548672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4597766265db4e84b281401e100c47f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:35.557761 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4597766265db4e84b281401e100c47f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:35.558494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4597766265db4e84b281401e100c47f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:35.575170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4597766265db4e84b281401e100c47f9'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:35.594869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1298/2590] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:35 2022] DELETE /identity/v3/users/4597766265db4e84b281401e100c47f9 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:35.596760 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:35.597299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:35.597299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:35.618193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:35.619767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:35.697000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1293/2591] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:35.706895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:35.708328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:35.771248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f34549-bbc0-44c9-9652-8e62c51f3b7c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:35.774890 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:35.775528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:35.775803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:35.776074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:35.787499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:35.789033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:35.799465 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1299/2592] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:36.070436 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:36.072091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:36.090198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e9b17a-c044-4991-b87c-1c5a81ceb80f None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:36.096499 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:36.097399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:36.097792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:36.098183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] PATH_INFO: `/v3/projects/d489a195eb55431ea8e591f485793061` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:36.099664 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d489a195eb55431ea8e591f485793061)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:36.102046 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:36.120556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:36.153923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd489a195eb55431ea8e591f485793061', 'request_id': 'req-14d1ee39-eac9-44fc-b2c4-614d77f4695c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.160164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd489a195eb55431ea8e591f485793061', 'request_id': 'req-14d1ee39-eac9-44fc-b2c4-614d77f4695c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.165696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd489a195eb55431ea8e591f485793061', 'request_id': 'req-14d1ee39-eac9-44fc-b2c4-614d77f4695c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.177986 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] The token cache is being invalidate because project d489a195eb55431ea8e591f485793061 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:36.178274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.178889 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.180401 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1294/2593] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:36 2022] DELETE /identity/v3/projects/d489a195eb55431ea8e591f485793061 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:36.184478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:36.194747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd56fa5-6184-4779-8b01-af68f403fb04 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:36.199312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:36.254024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:36.254658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:36.254938 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:36.255245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] PATH_INFO: `/v3/users/f4e1e94329b248b28f714120ef821532` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:36.256380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4e1e94329b248b28f714120ef821532)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:36.258300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:36.304291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4e1e94329b248b28f714120ef821532'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.304922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4e1e94329b248b28f714120ef821532'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.312060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4e1e94329b248b28f714120ef821532'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.312684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4e1e94329b248b28f714120ef821532'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.324746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4e1e94329b248b28f714120ef821532'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.336972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1300/2594] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:36 2022] DELETE /identity/v3/users/f4e1e94329b248b28f714120ef821532 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:36.626661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:36.628758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:36.652786 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d1ee39-eac9-44fc-b2c4-614d77f4695c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:36.717012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:36.717680 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:36.717987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:36.718261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] PATH_INFO: `/v3/projects/9f4ab9f71f434112895f78edfeeb6ac2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:36.720149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f4ab9f71f434112895f78edfeeb6ac2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:36.722194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:36.780191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f4ab9f71f434112895f78edfeeb6ac2', 'request_id': 'req-edb24622-9531-414a-9e3e-5812ab014781'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.791348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f4ab9f71f434112895f78edfeeb6ac2', 'request_id': 'req-edb24622-9531-414a-9e3e-5812ab014781'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.797525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f4ab9f71f434112895f78edfeeb6ac2', 'request_id': 'req-edb24622-9531-414a-9e3e-5812ab014781'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.816159 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] The token cache is being invalidate because project 9f4ab9f71f434112895f78edfeeb6ac2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:36.816625 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.817298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.818837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1295/2595] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:36 2022] DELETE /identity/v3/projects/9f4ab9f71f434112895f78edfeeb6ac2 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:36.822742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:36.824343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:36.839073 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d766be4c-8f0f-436d-971e-b01f1c2423c9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:36.904852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:36.905491 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:36.905768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:36.906041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] PATH_INFO: `/v3/users/f9e3d11620f74b66a3f871079cb81e0f` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:36.907337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9e3d11620f74b66a3f871079cb81e0f)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:36.909460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:36.944501 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:36.961818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9e3d11620f74b66a3f871079cb81e0f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.962453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9e3d11620f74b66a3f871079cb81e0f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.969539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9e3d11620f74b66a3f871079cb81e0f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.970147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9e3d11620f74b66a3f871079cb81e0f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.982681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ac31c12e-1a37-47f1-ae20-078a577faaf9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9e3d11620f74b66a3f871079cb81e0f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:36.997132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1301/2596] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:36 2022] DELETE /identity/v3/users/f9e3d11620f74b66a3f871079cb81e0f => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:43:37.303981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:37.305566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:37.320284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb24622-9531-414a-9e3e-5812ab014781 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:37.356167 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:37.388716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:37.389486 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.389826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.390163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] PATH_INFO: `/v3/projects/3173c879d2ae4d5abd8bbbdf4f1495a4` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.391877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3173c879d2ae4d5abd8bbbdf4f1495a4)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:37.393980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:37.441536 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3173c879d2ae4d5abd8bbbdf4f1495a4', 'request_id': 'req-cb01ca25-f787-4058-9eec-b45d8f53d9a6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:37.447787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3173c879d2ae4d5abd8bbbdf4f1495a4', 'request_id': 'req-cb01ca25-f787-4058-9eec-b45d8f53d9a6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:37.454259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3173c879d2ae4d5abd8bbbdf4f1495a4', 'request_id': 'req-cb01ca25-f787-4058-9eec-b45d8f53d9a6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:37.467043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] The token cache is being invalidate because project 3173c879d2ae4d5abd8bbbdf4f1495a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:37.467347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:37.467936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-cb01ca25-f787-4058-9eec-b45d8f53d9a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:37.470162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1296/2597] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:37 2022] DELETE /identity/v3/projects/3173c879d2ae4d5abd8bbbdf4f1495a4 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:37.477018 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bd07a9-e8e0-4d0a-82d5-cf482611ac7a None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.477307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bd07a9-e8e0-4d0a-82d5-cf482611ac7a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.477581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bd07a9-e8e0-4d0a-82d5-cf482611ac7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.498368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b5bd07a9-e8e0-4d0a-82d5-cf482611ac7a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:37.499698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b5bd07a9-e8e0-4d0a-82d5-cf482611ac7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:37.566170 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1302/2598] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:37.574466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29510438-7a16-4621-a2c0-3c74c1c00a7e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.574755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29510438-7a16-4621-a2c0-3c74c1c00a7e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.575052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29510438-7a16-4621-a2c0-3c74c1c00a7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.597620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-29510438-7a16-4621-a2c0-3c74c1c00a7e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:37.598856 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-29510438-7a16-4621-a2c0-3c74c1c00a7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:37.606829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1297/2599] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:37.611750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bd07a9-e8e0-4d0a-82d5-cf482611ac7a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:37.621817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bd07a9-e8e0-4d0a-82d5-cf482611ac7a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:37.626663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:37.627411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.627684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.627973 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.628622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:37.628913 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:37.630510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:37.643716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1303/2600] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:37.648313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29510438-7a16-4621-a2c0-3c74c1c00a7e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:37.660614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29510438-7a16-4621-a2c0-3c74c1c00a7e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:37.665314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:37.665913 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.666187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.666457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.667115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:37.669056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:37.700528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1298/2601] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:37 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:37.701624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:37.713740 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ceba75-db28-4bc2-9f32-6cad33fce601 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:37.780525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:37.781250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.781931 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.782379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.783372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:37.785380 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:37.807499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1304/2602] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:37 2022] POST /identity/v3/users => generated 287 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:37.811676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:37.823049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be36797d-5fe3-49b9-9d84-3d4a9146c68f admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:37.827842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:37.828429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.828830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.829103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.829739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:37.831388 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:37.841444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1299/2603] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:37.846058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:37.856049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1380cd4-37b4-4025-b080-5cefab7d1876 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:37.861354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7407443-0b89-4131-a3e0-9b9aa81a704c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:37.861948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7407443-0b89-4131-a3e0-9b9aa81a704c admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.862221 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7407443-0b89-4131-a3e0-9b9aa81a704c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.862492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7407443-0b89-4131-a3e0-9b9aa81a704c admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.863163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7407443-0b89-4131-a3e0-9b9aa81a704c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:37.864743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7407443-0b89-4131-a3e0-9b9aa81a704c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:37.879604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1305/2604] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:37.880838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:37.890930 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6202f51-7405-4591-82f1-2ce38e2c69bd admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:37.896369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:37.897062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.897342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.897614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] PATH_INFO: `/v3/projects/cd4a0dca907b485cbae314a7c477ee34/users/edcaf886b91b449badfce1d3fc6212c4/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.922929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd4a0dca907b485cbae314a7c477ee34, user_id=edcaf886b91b449badfce1d3fc6212c4, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:37.925720 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:37.941970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1300/2605] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:37 2022] PUT /identity/v3/projects/cd4a0dca907b485cbae314a7c477ee34/users/edcaf886b91b449badfce1d3fc6212c4/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:43:37.954296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17724ba7-4dbb-4ccb-85ea-82ae1a76ac00 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:37.954712 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17724ba7-4dbb-4ccb-85ea-82ae1a76ac00 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:37.955008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17724ba7-4dbb-4ccb-85ea-82ae1a76ac00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:37.982238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:37.986204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-17724ba7-4dbb-4ccb-85ea-82ae1a76ac00 None None] MFA Rules not processed for user `edcaf886b91b449badfce1d3fc6212c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:37.987470 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-17724ba7-4dbb-4ccb-85ea-82ae1a76ac00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:38.095944 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1306/2606] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:37 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:38.112109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:38.114253 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:38.234016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283643d2-c031-4b3e-9439-8afccd9495ee admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:38.254346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14098b7d-8a11-480e-be3d-4e2dd43d06d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:38.255607 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14098b7d-8a11-480e-be3d-4e2dd43d06d4 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:38.255999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14098b7d-8a11-480e-be3d-4e2dd43d06d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:38.256289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14098b7d-8a11-480e-be3d-4e2dd43d06d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:38.276471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14098b7d-8a11-480e-be3d-4e2dd43d06d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:38.278127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14098b7d-8a11-480e-be3d-4e2dd43d06d4 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:38.303799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1301/2607] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:38 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:38.336320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17724ba7-4dbb-4ccb-85ea-82ae1a76ac00 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:38.411864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17724ba7-4dbb-4ccb-85ea-82ae1a76ac00 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:38.416784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:38.417496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:38.417771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:38.418042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] PATH_INFO: `/v3/users/edcaf886b91b449badfce1d3fc6212c4` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:38.419189 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=edcaf886b91b449badfce1d3fc6212c4)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:38.421109 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:38.490197 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'edcaf886b91b449badfce1d3fc6212c4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:38.491305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'edcaf886b91b449badfce1d3fc6212c4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:38.500268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'edcaf886b91b449badfce1d3fc6212c4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:38.501534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'edcaf886b91b449badfce1d3fc6212c4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:38.514052 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'edcaf886b91b449badfce1d3fc6212c4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:38.528619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1307/2608] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:38 2022] DELETE /identity/v3/users/edcaf886b91b449badfce1d3fc6212c4 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:38.533344 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:38.533635 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:38.533905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:38.543478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:38.558949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:38.560761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:38.629201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1302/2609] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:38.637649 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:38.639970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:38.669789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:38.716261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c110e45b-7869-4cac-ba14-b4ecf47ecbcb admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:38.722942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dc1e61-bde9-434d-8c36-cfd6fca54587 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:38.723600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dc1e61-bde9-434d-8c36-cfd6fca54587 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:38.724239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dc1e61-bde9-434d-8c36-cfd6fca54587 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:38.724239 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dc1e61-bde9-434d-8c36-cfd6fca54587 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:38.736214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dc1e61-bde9-434d-8c36-cfd6fca54587 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:38.737759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dc1e61-bde9-434d-8c36-cfd6fca54587 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:38.747942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1308/2610] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:39.043520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.047466 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:39.065259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d0a72-71df-4b10-84c9-53aa7c8fe7b1 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.070394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.071066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.071347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.072177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] PATH_INFO: `/v3/projects/cd4a0dca907b485cbae314a7c477ee34` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.073690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd4a0dca907b485cbae314a7c477ee34)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.076346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.149805 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd4a0dca907b485cbae314a7c477ee34', 'request_id': 'req-8210e15e-55df-4d1b-baaf-f1399ec5899a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:39.156375 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd4a0dca907b485cbae314a7c477ee34', 'request_id': 'req-8210e15e-55df-4d1b-baaf-f1399ec5899a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:39.163746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd4a0dca907b485cbae314a7c477ee34', 'request_id': 'req-8210e15e-55df-4d1b-baaf-f1399ec5899a'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:39.179878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] The token cache is being invalidate because project cd4a0dca907b485cbae314a7c477ee34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:39.180405 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:39.181115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8210e15e-55df-4d1b-baaf-f1399ec5899a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:39.182837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1303/2611] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:39 2022] DELETE /identity/v3/projects/cd4a0dca907b485cbae314a7c477ee34 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:39.188779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6137fec1-c429-4fb0-8d7d-ab9bf1dec9a3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.189266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6137fec1-c429-4fb0-8d7d-ab9bf1dec9a3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.189376 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6137fec1-c429-4fb0-8d7d-ab9bf1dec9a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.211112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6137fec1-c429-4fb0-8d7d-ab9bf1dec9a3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:39.219678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6137fec1-c429-4fb0-8d7d-ab9bf1dec9a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:39.295284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1309/2612] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:39.303682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc223a9-25a6-45f1-b35d-ce2853585c07 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.303971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc223a9-25a6-45f1-b35d-ce2853585c07 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.304244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc223a9-25a6-45f1-b35d-ce2853585c07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.332449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4fc223a9-25a6-45f1-b35d-ce2853585c07 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:39.338508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4fc223a9-25a6-45f1-b35d-ce2853585c07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:39.348317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1304/2613] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:39.352691 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6137fec1-c429-4fb0-8d7d-ab9bf1dec9a3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.363101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6137fec1-c429-4fb0-8d7d-ab9bf1dec9a3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.367765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.368329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.368607 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.368906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.369812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:39.370117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.371772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.384822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1310/2614] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:39.389191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc223a9-25a6-45f1-b35d-ce2853585c07 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.398738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc223a9-25a6-45f1-b35d-ce2853585c07 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.403545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.404155 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.404429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.404704 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.405332 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.409239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.442907 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1305/2615] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:39 2022] POST /identity/v3/projects => generated 371 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:39.446708 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.457309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbd308-b0f9-49f7-a2d7-d3d8f46c23ca admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.516947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.517704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.517983 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.518282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.518886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.520934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.549168 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1311/2616] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:39 2022] POST /identity/v3/users => generated 286 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:39.550521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.561584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd85a38-a994-4749-a4dd-a95e2b974305 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.567066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.567669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.567993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.568348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.568994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.570602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.581631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1306/2617] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:39.585700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.598629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea73cd63-17de-4cc7-bc7f-261b9be1f243 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.604748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdefca38-62de-4a2c-a9c8-fef011b7a1ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.605360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdefca38-62de-4a2c-a9c8-fef011b7a1ec admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.605641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdefca38-62de-4a2c-a9c8-fef011b7a1ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.605916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdefca38-62de-4a2c-a9c8-fef011b7a1ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.606563 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdefca38-62de-4a2c-a9c8-fef011b7a1ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.608368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdefca38-62de-4a2c-a9c8-fef011b7a1ec admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.621244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1312/2618] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:39 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:39.625767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.637230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9791e7e2-801e-4ec6-bbb2-d4aa833037ac admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.645287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.646026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.646409 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.646770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] PATH_INFO: `/v3/projects/0a6698bc3af34041bd01a5d5abefb11c/users/040bc65ba7f94c14999a98266ff0ce7f/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.664408 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a6698bc3af34041bd01a5d5abefb11c, user_id=040bc65ba7f94c14999a98266ff0ce7f, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.667114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.686696 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1307/2619] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:39 2022] PUT /identity/v3/projects/0a6698bc3af34041bd01a5d5abefb11c/users/040bc65ba7f94c14999a98266ff0ce7f/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:43:39.691687 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c93caa5-c208-4ed5-9768-4a896400f4ae None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.691975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c93caa5-c208-4ed5-9768-4a896400f4ae None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.692340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c93caa5-c208-4ed5-9768-4a896400f4ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.716383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1c93caa5-c208-4ed5-9768-4a896400f4ae None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:39.717636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1c93caa5-c208-4ed5-9768-4a896400f4ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:39.749602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:39.792303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1313/2620] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:39.803069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:39.806006 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:39.849243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:39.900442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1f8d6-59f0-4d80-81e7-50324367047b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:39.906008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81873a9-2779-4bad-8e18-f27ed2eecd85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:39.907703 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81873a9-2779-4bad-8e18-f27ed2eecd85 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:39.908362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81873a9-2779-4bad-8e18-f27ed2eecd85 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:39.908796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81873a9-2779-4bad-8e18-f27ed2eecd85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:39.921201 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81873a9-2779-4bad-8e18-f27ed2eecd85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:39.923508 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81873a9-2779-4bad-8e18-f27ed2eecd85 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:39.935227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1308/2621] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:42.400374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c93caa5-c208-4ed5-9768-4a896400f4ae None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:42.410591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c93caa5-c208-4ed5-9768-4a896400f4ae None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:42.415489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:42.416213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:42.416626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:42.417023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:42.419094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:42.434303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:42.435986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:42.446729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1314/2622] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:43:44.373917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff63d29-e256-477a-b921-ed7f93861323 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:44.374384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff63d29-e256-477a-b921-ed7f93861323 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:44.374586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff63d29-e256-477a-b921-ed7f93861323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:44.405938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-8ff63d29-e256-477a-b921-ed7f93861323 None None] MFA Rules not processed for user `040bc65ba7f94c14999a98266ff0ce7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:44.408047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8ff63d29-e256-477a-b921-ed7f93861323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:44.495892 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1309/2623] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:44 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:44.499980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:44.515008 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a182270b-9543-4f24-9d6d-dc673a764a57 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:44.521832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:44.523214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:44.523665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:44.524116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:44.524939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:44.527013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:44.571226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1315/2624] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:44 2022] POST /identity/v3/projects => generated 373 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:44.575843 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff63d29-e256-477a-b921-ed7f93861323 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:44.597651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff63d29-e256-477a-b921-ed7f93861323 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:44.691523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:44.692751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:44.693037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:44.693371 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:44.694054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:44.696211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:44.730744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1310/2625] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:44 2022] POST /identity/v3/users => generated 293 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:44.732061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:44.755654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a416b9-0b65-4482-a235-7808ec9c30d2 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:44.762770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:44.763717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:44.778159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:44.779937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:44.780832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:44.786220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:44.799215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1316/2626] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:44 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:44.802842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:44.816127 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987299d-907f-43aa-916c-61d754d05cce admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:44.825198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:44.826051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:44.826258 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:44.826532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] PATH_INFO: `/v3/projects/794a333acb874a3784e504154d321157/users/b466a7db4d11416fbba073555a210d0d/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:44.853166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=794a333acb874a3784e504154d321157, user_id=b466a7db4d11416fbba073555a210d0d, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:44.856058 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:44.884755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1311/2627] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:44 2022] PUT /identity/v3/projects/794a333acb874a3784e504154d321157/users/b466a7db4d11416fbba073555a210d0d/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:44.885973 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:44.909604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55101675-0c0b-4f86-a34e-43fdb0f7da36 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:45.020528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:45.022044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:45.022620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:45.023800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:45.024672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:45.026861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:45.053010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1317/2628] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:44 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:45.056849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:45.068647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ede7695-898d-4d02-bc8d-ae57fa9746ca admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:45.074041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:45.074796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:45.075183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:45.075784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] PATH_INFO: `/v3/projects/794a333acb874a3784e504154d321157/users/b466a7db4d11416fbba073555a210d0d/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:45.079282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=794a333acb874a3784e504154d321157, user_id=b466a7db4d11416fbba073555a210d0d, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:45.081126 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:45.104739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1312/2629] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:45 2022] PUT /identity/v3/projects/794a333acb874a3784e504154d321157/users/b466a7db4d11416fbba073555a210d0d/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:46.738838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:46.742000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:46.767930 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b91cd51-bd44-4c9f-9d0a-b9c6fb07d4cc admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:46.852559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:46.853200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:46.853665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:46.853937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] PATH_INFO: `/v3/users/d574235b791e485aa64130c599539945` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:46.855549 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=d574235b791e485aa64130c599539945)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:46.857673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:46.882671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:46.927769 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd574235b791e485aa64130c599539945'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:46.928631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd574235b791e485aa64130c599539945'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:46.943581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd574235b791e485aa64130c599539945'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:46.944741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd574235b791e485aa64130c599539945'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:46.963288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e2cdf9a9-6b22-4aec-8fb2-1011d01106ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd574235b791e485aa64130c599539945'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:46.980318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1318/2630] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:46 2022] DELETE /identity/v3/users/d574235b791e485aa64130c599539945 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:47.250193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:47.255777 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:47.282753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d50470-8688-44b0-89a4-e03aceef83a5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:47.286923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:47.378148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:47.379176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:47.379710 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:47.380135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] PATH_INFO: `/v3/projects/594a13a702cf47898a48ab4fd6789359` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:47.381585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=594a13a702cf47898a48ab4fd6789359)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:47.383980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:47.450321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '594a13a702cf47898a48ab4fd6789359', 'request_id': 'req-ebdae6bb-698f-4694-943d-47798b1c16c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:47.460523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '594a13a702cf47898a48ab4fd6789359', 'request_id': 'req-ebdae6bb-698f-4694-943d-47798b1c16c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:47.470222 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '594a13a702cf47898a48ab4fd6789359', 'request_id': 'req-ebdae6bb-698f-4694-943d-47798b1c16c8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:47.488369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] The token cache is being invalidate because project 594a13a702cf47898a48ab4fd6789359 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:47.488951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:47.492277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ebdae6bb-698f-4694-943d-47798b1c16c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:47.498224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1313/2631] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:47 2022] DELETE /identity/v3/projects/594a13a702cf47898a48ab4fd6789359 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:47.501006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512221e8-3d86-4e47-8fc3-7abbd01fb753 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:47.501293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512221e8-3d86-4e47-8fc3-7abbd01fb753 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:47.501755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512221e8-3d86-4e47-8fc3-7abbd01fb753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:47.545244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-512221e8-3d86-4e47-8fc3-7abbd01fb753 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:47.546331 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-512221e8-3d86-4e47-8fc3-7abbd01fb753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:47.627204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1319/2632] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:47.635190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c4770-da30-43ae-adad-0f354bcb6dcb None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:47.635478 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c4770-da30-43ae-adad-0f354bcb6dcb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:47.635740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c4770-da30-43ae-adad-0f354bcb6dcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:47.657612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0d0c4770-da30-43ae-adad-0f354bcb6dcb None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:47.660333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0d0c4770-da30-43ae-adad-0f354bcb6dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:47.668393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1314/2633] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:43:47.673011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512221e8-3d86-4e47-8fc3-7abbd01fb753 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:47.683359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512221e8-3d86-4e47-8fc3-7abbd01fb753 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:47.688559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:47.689183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:47.689704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:47.689985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:47.690646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:47.690947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:47.692909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:47.714287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1320/2634] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:47.715602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c4770-da30-43ae-adad-0f354bcb6dcb None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:47.730722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c4770-da30-43ae-adad-0f354bcb6dcb None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:47.743141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:47.743888 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:47.744261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:47.744826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:47.745658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:47.747890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:47.815505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1315/2635] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:47 2022] POST /identity/v3/projects => generated 399 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:47.819814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:47.833650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02983da0-d7c0-4547-a1ea-5adf99334659 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:47.918143 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:47.920115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:47.920391 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:47.920841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:47.923275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:47.925723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:47.964342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1321/2636] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:47 2022] POST /identity/v3/users => generated 300 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:47.968392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:47.981308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b5bf7b-2bbf-464a-bd60-b4acd8171f53 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:47.988596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:47.989202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:47.989489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:47.989909 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:47.990564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:47.992354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:48.007906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1316/2637] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:47 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:48.009300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:48.024872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6d8976-f07b-4413-8ae5-6bd8b699f1f8 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:48.041687 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6696a533-46c0-4520-a370-7a2d3d8db5fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:48.042462 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6696a533-46c0-4520-a370-7a2d3d8db5fc admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:48.042752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6696a533-46c0-4520-a370-7a2d3d8db5fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:48.043174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6696a533-46c0-4520-a370-7a2d3d8db5fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:48.043856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6696a533-46c0-4520-a370-7a2d3d8db5fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:48.045614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6696a533-46c0-4520-a370-7a2d3d8db5fc admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:48.065644 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1322/2638] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:48 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:48.067153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:48.099011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c69eaf7-5cb9-4cd9-a830-6b5b2076d074 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:48.112307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:48.115331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:48.117238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:48.118679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] PATH_INFO: `/v3/projects/f4052bd209fe40c383417fc8b888e17c/users/9779cc9397994ca5828dfb3a23faad59/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:48.152325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4052bd209fe40c383417fc8b888e17c, user_id=9779cc9397994ca5828dfb3a23faad59, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:48.155356 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:48.176947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1317/2639] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:48 2022] PUT /identity/v3/projects/f4052bd209fe40c383417fc8b888e17c/users/9779cc9397994ca5828dfb3a23faad59/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:48.190943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc45cb56-7a4e-4bc1-9387-9789c2dd8ce7 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:48.191576 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc45cb56-7a4e-4bc1-9387-9789c2dd8ce7 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:48.191958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc45cb56-7a4e-4bc1-9387-9789c2dd8ce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:48.212487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:48.239162 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-dc45cb56-7a4e-4bc1-9387-9789c2dd8ce7 None None] MFA Rules not processed for user `9779cc9397994ca5828dfb3a23faad59`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:48.242017 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-dc45cb56-7a4e-4bc1-9387-9789c2dd8ce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:48.367111 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1323/2640] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:48 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:48.370449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:48.427263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:48.492313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc6cb8e-45d9-406e-9fc4-d9da55aaa624 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:48.507822 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:48.509669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:48.510146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:48.510251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:48.510851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:48.512810 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:48.568735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1318/2641] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:48 2022] POST /identity/v3/projects => generated 397 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:43:48.570108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc45cb56-7a4e-4bc1-9387-9789c2dd8ce7 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:48.586135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc45cb56-7a4e-4bc1-9387-9789c2dd8ce7 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:48.683419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:48.684340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:48.684722 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:48.685627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:48.686453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:48.701365 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:48.757701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1324/2642] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:48 2022] POST /identity/v3/users => generated 305 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:48.761693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:48.782005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab590741-de60-4068-bc99-8eed1b76eef3 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:48.793891 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:48.794528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:48.794801 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:48.795100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:48.795758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:48.802049 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:48.847643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1319/2643] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:48 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:43:48.852244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:48.865648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced3d4b3-38cb-4714-aa30-b6fcd9854b3e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:48.871961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:48.872811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:48.873088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:48.873359 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] PATH_INFO: `/v3/projects/dac0cf152545448bad420a0b6c53713c/users/79f68a1a95544ea995e1685ee1b6c7d6/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:48.899862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dac0cf152545448bad420a0b6c53713c, user_id=79f68a1a95544ea995e1685ee1b6c7d6, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:48.903179 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:48.928924 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1325/2644] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:48 2022] PUT /identity/v3/projects/dac0cf152545448bad420a0b6c53713c/users/79f68a1a95544ea995e1685ee1b6c7d6/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:43:48.932961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:48.946821 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0601910-6f72-440d-9953-8bcc12d68727 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:49.023316 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d075383a-e1a0-45bf-b9ca-5f8ca08fd2b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:49.024298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d075383a-e1a0-45bf-b9ca-5f8ca08fd2b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:49.024767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d075383a-e1a0-45bf-b9ca-5f8ca08fd2b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:49.025113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d075383a-e1a0-45bf-b9ca-5f8ca08fd2b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:49.026312 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d075383a-e1a0-45bf-b9ca-5f8ca08fd2b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:49.028289 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d075383a-e1a0-45bf-b9ca-5f8ca08fd2b7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:49.043840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1320/2645] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:48 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:49.048660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:49.063355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d980edb-9dcd-4a7a-97ae-4c9b9226eb92 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:49.069324 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:49.070609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:49.070991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:49.071378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] PATH_INFO: `/v3/projects/dac0cf152545448bad420a0b6c53713c/users/79f68a1a95544ea995e1685ee1b6c7d6/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:49.076886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dac0cf152545448bad420a0b6c53713c, user_id=79f68a1a95544ea995e1685ee1b6c7d6, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:49.080961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:49.103611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1326/2646] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:49 2022] PUT /identity/v3/projects/dac0cf152545448bad420a0b6c53713c/users/79f68a1a95544ea995e1685ee1b6c7d6/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:49.115306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7a7cf-7f98-4b2e-82ce-c26726b8835a None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:49.115589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7a7cf-7f98-4b2e-82ce-c26726b8835a None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:49.115890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7a7cf-7f98-4b2e-82ce-c26726b8835a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:49.160220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-0cc7a7cf-7f98-4b2e-82ce-c26726b8835a None None] MFA Rules not processed for user `79f68a1a95544ea995e1685ee1b6c7d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:49.160943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0cc7a7cf-7f98-4b2e-82ce-c26726b8835a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:49.327067 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1321/2647] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:49 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:49.341213 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:49.343815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:49.466659 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:49.474107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cd19ca-3bcd-4d9f-8222-1bcdfa761765 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:49.479642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc1837c-d250-4666-8f0f-ae39c2a7e2bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:49.481337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc1837c-d250-4666-8f0f-ae39c2a7e2bb service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:49.481858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc1837c-d250-4666-8f0f-ae39c2a7e2bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:49.482354 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc1837c-d250-4666-8f0f-ae39c2a7e2bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:49.494245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc1837c-d250-4666-8f0f-ae39c2a7e2bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:49.496505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc1837c-d250-4666-8f0f-ae39c2a7e2bb service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:49.524028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1327/2648] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:49 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:49.624412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7a7cf-7f98-4b2e-82ce-c26726b8835a None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:49.638341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7a7cf-7f98-4b2e-82ce-c26726b8835a None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:49.678093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:49.678840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:49.679172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:49.679872 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:49.729882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:49.794295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:49.796465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:49.824293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1322/2649] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:49 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:50.908726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5377a176-a526-4abd-8cd2-484af3b6cb26 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:50.909657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5377a176-a526-4abd-8cd2-484af3b6cb26 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:50.910383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5377a176-a526-4abd-8cd2-484af3b6cb26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:50.951071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5377a176-a526-4abd-8cd2-484af3b6cb26 None None] MFA Rules not processed for user `b466a7db4d11416fbba073555a210d0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:50.954879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5377a176-a526-4abd-8cd2-484af3b6cb26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:51.049285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1328/2650] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:50 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:43:51.064858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:51.067313 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:51.177007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8517da7c-33a2-42d7-8f7f-501e9c03c52f service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:51.184666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:51.185378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:51.185724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:51.186069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:51.188940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:51.268593 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:51.270338 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:51.280052 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1323/2651] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:51 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:52.335323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a0ae7-e81a-48a7-acfd-5320eab56a32 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:52.335869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a0ae7-e81a-48a7-acfd-5320eab56a32 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:52.336025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341a0ae7-e81a-48a7-acfd-5320eab56a32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:52.356161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-341a0ae7-e81a-48a7-acfd-5320eab56a32 None None] MFA Rules not processed for user `040bc65ba7f94c14999a98266ff0ce7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:52.357351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-341a0ae7-e81a-48a7-acfd-5320eab56a32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:52.366106 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1329/2652] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:52 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:57.306219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:57.372418 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f7ec0-ff2b-4148-bdc8-d89495d96471 service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:57.377382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:57.378015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:57.378294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:57.378570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] PATH_INFO: `/v3/users/9779cc9397994ca5828dfb3a23faad59` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:57.379741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] RBAC: Authorizing `identity:delete_user(user_id=9779cc9397994ca5828dfb3a23faad59)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:57.381662 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:57.388075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=100 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:57.433606 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9779cc9397994ca5828dfb3a23faad59'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:57.434229 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9779cc9397994ca5828dfb3a23faad59'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:57.441404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9779cc9397994ca5828dfb3a23faad59'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:57.460452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9779cc9397994ca5828dfb3a23faad59'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:57.460452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9779cc9397994ca5828dfb3a23faad59'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:57.472601 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1324/2653] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:57 2022] DELETE /identity/v3/users/9779cc9397994ca5828dfb3a23faad59 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:57.477717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:57.478253 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:57.478355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:57.498755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:57.500063 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:57.569121 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1330/2654] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:43:57.580723 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:57.582342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:57.648487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c355c97b-9566-4b90-be5a-81576cd4d56c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:57.652434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:57.653140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:57.653473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:57.653744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:57.666214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:57.666214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:57.676514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1325/2655] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:43:57.958306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:57.960218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:57.973812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c367ad69-4d9b-49dc-95d9-1ac1db2a3d78 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:57.978697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:57.979361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:57.979643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:57.979910 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] PATH_INFO: `/v3/projects/f4052bd209fe40c383417fc8b888e17c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:57.981493 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4052bd209fe40c383417fc8b888e17c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:57.983655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:58.031541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4052bd209fe40c383417fc8b888e17c', 'request_id': 'req-e299d4ca-646b-4446-afed-625297044217'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.037619 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4052bd209fe40c383417fc8b888e17c', 'request_id': 'req-e299d4ca-646b-4446-afed-625297044217'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.043876 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4052bd209fe40c383417fc8b888e17c', 'request_id': 'req-e299d4ca-646b-4446-afed-625297044217'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.046362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:58.059578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] The token cache is being invalidate because project f4052bd209fe40c383417fc8b888e17c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:58.060044 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.060664 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.062159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1331/2656] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:57 2022] DELETE /identity/v3/projects/f4052bd209fe40c383417fc8b888e17c => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:58.066158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:58.075896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3627a79-3844-46ca-a492-882eee5bd02b service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:58.132586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:58.133103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:58.133374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:58.133639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] PATH_INFO: `/v3/users/79f68a1a95544ea995e1685ee1b6c7d6` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:58.134744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=79f68a1a95544ea995e1685ee1b6c7d6)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:58.136708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:58.171698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:58.192746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79f68a1a95544ea995e1685ee1b6c7d6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.193399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79f68a1a95544ea995e1685ee1b6c7d6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.200431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79f68a1a95544ea995e1685ee1b6c7d6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.213146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79f68a1a95544ea995e1685ee1b6c7d6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.213997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-97ec5481-79ed-4e4f-ae29-ddaae9b5cd2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79f68a1a95544ea995e1685ee1b6c7d6'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.228320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1326/2657] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:43:58 2022] DELETE /identity/v3/users/79f68a1a95544ea995e1685ee1b6c7d6 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:58.510013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:58.511666 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:58.528865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e299d4ca-646b-4446-afed-625297044217 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:58.590527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:58.591187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:58.591469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:58.591742 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] PATH_INFO: `/v3/projects/dac0cf152545448bad420a0b6c53713c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:58.592870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] RBAC: Authorizing `identity:delete_project(project_id=dac0cf152545448bad420a0b6c53713c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:58.595353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:58.645214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dac0cf152545448bad420a0b6c53713c', 'request_id': 'req-82a89f36-c50c-44c6-85c0-9ccfd586794a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.650634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dac0cf152545448bad420a0b6c53713c', 'request_id': 'req-82a89f36-c50c-44c6-85c0-9ccfd586794a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.660707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dac0cf152545448bad420a0b6c53713c', 'request_id': 'req-82a89f36-c50c-44c6-85c0-9ccfd586794a'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.672754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] The token cache is being invalidate because project dac0cf152545448bad420a0b6c53713c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:43:58.674219 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.674867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-82a89f36-c50c-44c6-85c0-9ccfd586794a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:43:58.676389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1332/2658] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:43:58 2022] DELETE /identity/v3/projects/dac0cf152545448bad420a0b6c53713c => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:58.681217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1439a357-94af-4935-bee7-69f26d638811 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:58.681708 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1439a357-94af-4935-bee7-69f26d638811 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:58.681851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1439a357-94af-4935-bee7-69f26d638811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:58.703471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-1439a357-94af-4935-bee7-69f26d638811 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:58.705416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1439a357-94af-4935-bee7-69f26d638811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:58.768866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1327/2659] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:58.778177 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d8e21c-cfa4-427a-933b-64e576c0e39c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:58.778460 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d8e21c-cfa4-427a-933b-64e576c0e39c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:58.778794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d8e21c-cfa4-427a-933b-64e576c0e39c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:58.801353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-72d8e21c-cfa4-427a-933b-64e576c0e39c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:58.802582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-72d8e21c-cfa4-427a-933b-64e576c0e39c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:58.810527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1333/2660] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:58.814751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1439a357-94af-4935-bee7-69f26d638811 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:58.824999 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1439a357-94af-4935-bee7-69f26d638811 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:58.830189 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:58.830849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:58.831306 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:58.831773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:58.832496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:43:58.832879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:58.834546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:58.848693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1328/2661] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:43:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:43:58.852821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d8e21c-cfa4-427a-933b-64e576c0e39c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:58.863371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d8e21c-cfa4-427a-933b-64e576c0e39c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:58.868075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:58.868684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:58.868958 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:58.869228 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:58.870118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:58.872067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:58.910244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1334/2662] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:58 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:43:58.911009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:58.925207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82e5cea-3771-4322-856b-f50453d537be admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:58.988673 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:58.989286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:58.989566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:58.989836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:58.990615 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:58.992837 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.019245 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1329/2663] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:58 2022] POST /identity/v3/users => generated 291 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:59.023426 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.036116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b94afb-8676-4c29-8b33-72db34edc397 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.040908 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.041511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.041788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.042060 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.042707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.044342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.059033 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1335/2664] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:59 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:59.062884 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.073039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e284a-5f8c-46dc-b3cd-d0d409e378f8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.078028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5746eaa-c6fb-4df5-b80f-75dd20c836a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.078650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5746eaa-c6fb-4df5-b80f-75dd20c836a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.078931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5746eaa-c6fb-4df5-b80f-75dd20c836a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.079744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5746eaa-c6fb-4df5-b80f-75dd20c836a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.080384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5746eaa-c6fb-4df5-b80f-75dd20c836a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.081955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5746eaa-c6fb-4df5-b80f-75dd20c836a3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.095475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1330/2665] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:59.099529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.109413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3e0b8-11da-414a-9d91-fc054e952e07 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.114705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.115416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.115697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.116092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] PATH_INFO: `/v3/projects/223a07a9b1c9402aa59fb9efe7bfde9c/users/cc1a601f698749519343aef845df630f/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.134409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] RBAC: Authorizing `identity:create_grant(project_id=223a07a9b1c9402aa59fb9efe7bfde9c, user_id=cc1a601f698749519343aef845df630f, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.137309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.154024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1336/2666] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:59 2022] PUT /identity/v3/projects/223a07a9b1c9402aa59fb9efe7bfde9c/users/cc1a601f698749519343aef845df630f/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:59.163706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf1201-f3a1-491d-a724-dec1cb097ff9 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.163998 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf1201-f3a1-491d-a724-dec1cb097ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.164269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf1201-f3a1-491d-a724-dec1cb097ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.185712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:59.199280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-72cf1201-f3a1-491d-a724-dec1cb097ff9 None None] MFA Rules not processed for user `cc1a601f698749519343aef845df630f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:59.201840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-72cf1201-f3a1-491d-a724-dec1cb097ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:59.293279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1331/2667] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:59 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:59.298590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.319260 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:59.361740 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de76c60-24a6-46e0-acd2-af66ad3f6438 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.366347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.366940 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.367244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.367516 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.368147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.370301 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.397366 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1337/2668] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:43:59 2022] POST /identity/v3/projects => generated 381 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:59.401687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf1201-f3a1-491d-a724-dec1cb097ff9 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.413489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cf1201-f3a1-491d-a724-dec1cb097ff9 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.474587 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.475235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.475528 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.475800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.476440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.478386 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.505248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1332/2669] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:43:59 2022] POST /identity/v3/users => generated 297 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:43:59.506618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.517410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85789e0c-59de-4ea8-893c-36c028a53413 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.522496 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.523139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.523449 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.523715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.524438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.526210 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.538993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1338/2670] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:59.543110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.554463 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae435454-3c5d-4c22-b947-9baf612e4a18 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.559426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.560069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.560349 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.560624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] PATH_INFO: `/v3/projects/0a9bff518ea94c908dc4903145b0841b/users/c2289f6c111540d7a716abbb0703a4d5/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.577961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a9bff518ea94c908dc4903145b0841b, user_id=c2289f6c111540d7a716abbb0703a4d5, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.581374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.596462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1333/2671] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:59 2022] PUT /identity/v3/projects/0a9bff518ea94c908dc4903145b0841b/users/c2289f6c111540d7a716abbb0703a4d5/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:59.600780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.611385 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d8ea96-7294-4d74-90ac-97815b28abee admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.669801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b22ffb3-4454-42ad-919d-872b46bdc3e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.670392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b22ffb3-4454-42ad-919d-872b46bdc3e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.670671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b22ffb3-4454-42ad-919d-872b46bdc3e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.670944 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b22ffb3-4454-42ad-919d-872b46bdc3e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.671622 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b22ffb3-4454-42ad-919d-872b46bdc3e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.673463 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b22ffb3-4454-42ad-919d-872b46bdc3e3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.683428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1339/2672] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:43:59 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:43:59.688071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.698594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541ed137-11a1-4de8-a1f6-dac4a6f93249 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.703653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.705103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.705438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.705751 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] PATH_INFO: `/v3/projects/0a9bff518ea94c908dc4903145b0841b/users/c2289f6c111540d7a716abbb0703a4d5/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.708004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a9bff518ea94c908dc4903145b0841b, user_id=c2289f6c111540d7a716abbb0703a4d5, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.710818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.722396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1334/2673] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:43:59 2022] PUT /identity/v3/projects/0a9bff518ea94c908dc4903145b0841b/users/c2289f6c111540d7a716abbb0703a4d5/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:43:59.732554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f554d-94aa-47e8-a06b-c6140a54a85d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.732838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f554d-94aa-47e8-a06b-c6140a54a85d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.733112 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f554d-94aa-47e8-a06b-c6140a54a85d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.765432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-a14f554d-94aa-47e8-a06b-c6140a54a85d None None] MFA Rules not processed for user `c2289f6c111540d7a716abbb0703a4d5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:43:59.767202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a14f554d-94aa-47e8-a06b-c6140a54a85d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:59.836771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1340/2674] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:43:59 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:43:59.848178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:43:59.849833 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:43:59.911264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7c7c4a-6490-4498-b5ba-8ad00330fdd7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:43:59.915634 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:43:59.916250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:43:59.916524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:43:59.916792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:43:59.924136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:43:59.926666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:43:59.928749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:43:59.939971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1335/2675] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:43:59 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:00.052199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f554d-94aa-47e8-a06b-c6140a54a85d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:00.062041 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f554d-94aa-47e8-a06b-c6140a54a85d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:00.066103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20495603-6a45-402b-8dbc-2644663a067b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:00.066700 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20495603-6a45-402b-8dbc-2644663a067b service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:00.066993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20495603-6a45-402b-8dbc-2644663a067b service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:00.067269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20495603-6a45-402b-8dbc-2644663a067b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:00.079812 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:00.127887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20495603-6a45-402b-8dbc-2644663a067b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:00.129361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20495603-6a45-402b-8dbc-2644663a067b service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:00.139471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1341/2676] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:00 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:02.947834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:03.016024 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bda784-dd86-4ea8-9547-ab4ae0da61e5 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:03.022028 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:03.022650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:03.022925 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:03.023227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] PATH_INFO: `/v3/users/cc1a601f698749519343aef845df630f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:03.024347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc1a601f698749519343aef845df630f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:03.026242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:03.077011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc1a601f698749519343aef845df630f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.077666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc1a601f698749519343aef845df630f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.085869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc1a601f698749519343aef845df630f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.098571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc1a601f698749519343aef845df630f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.099329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc1a601f698749519343aef845df630f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.113616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1336/2677] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:02 2022] DELETE /identity/v3/users/cc1a601f698749519343aef845df630f => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:03.118353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:03.118835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:03.118929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:03.140943 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:03.142152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:03.207353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1342/2678] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:03.217464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:03.218983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:03.290744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6339a-1d56-45c9-a568-d2d330606de1 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:03.294520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:03.295182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:03.295460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:03.295729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:03.306404 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:03.307854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:03.317479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1337/2679] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:03.605984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:03.607442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:03.621602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e431c9e-5084-41d6-a573-a206b80d102c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:03.626554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:03.627658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:03.627971 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:03.628271 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] PATH_INFO: `/v3/projects/223a07a9b1c9402aa59fb9efe7bfde9c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:03.629506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] RBAC: Authorizing `identity:delete_project(project_id=223a07a9b1c9402aa59fb9efe7bfde9c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:03.631393 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:03.695418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '223a07a9b1c9402aa59fb9efe7bfde9c', 'request_id': 'req-37839774-f71e-42c9-b882-21602f6e5604'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.700882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '223a07a9b1c9402aa59fb9efe7bfde9c', 'request_id': 'req-37839774-f71e-42c9-b882-21602f6e5604'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.707581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '223a07a9b1c9402aa59fb9efe7bfde9c', 'request_id': 'req-37839774-f71e-42c9-b882-21602f6e5604'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.720185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] The token cache is being invalidate because project 223a07a9b1c9402aa59fb9efe7bfde9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:03.720464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.721075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.722772 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1343/2680] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:03 2022] DELETE /identity/v3/projects/223a07a9b1c9402aa59fb9efe7bfde9c => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:03.726722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:03.736420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63470291-271c-4663-8ea7-4f7c65d3d086 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:03.748740 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:03.795961 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:03.796729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:03.797061 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:03.797394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] PATH_INFO: `/v3/users/c2289f6c111540d7a716abbb0703a4d5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:03.798819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2289f6c111540d7a716abbb0703a4d5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:03.800812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:03.861498 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2289f6c111540d7a716abbb0703a4d5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.862146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2289f6c111540d7a716abbb0703a4d5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.869300 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2289f6c111540d7a716abbb0703a4d5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.881819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2289f6c111540d7a716abbb0703a4d5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.882496 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-c4da9029-6ab0-4a42-b586-8363d05fd2f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2289f6c111540d7a716abbb0703a4d5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:03.898929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1338/2681] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:03 2022] DELETE /identity/v3/users/c2289f6c111540d7a716abbb0703a4d5 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:04.189361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.191640 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:04.204396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37839774-f71e-42c9-b882-21602f6e5604 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.243829 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:04.270145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.270778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.271097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.271963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] PATH_INFO: `/v3/projects/0a9bff518ea94c908dc4903145b0841b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.273088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a9bff518ea94c908dc4903145b0841b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:04.275629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:04.324133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a9bff518ea94c908dc4903145b0841b', 'request_id': 'req-25a31519-00f0-445e-9926-d4076321e231'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:04.329898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a9bff518ea94c908dc4903145b0841b', 'request_id': 'req-25a31519-00f0-445e-9926-d4076321e231'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:04.337698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a9bff518ea94c908dc4903145b0841b', 'request_id': 'req-25a31519-00f0-445e-9926-d4076321e231'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:04.358425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] The token cache is being invalidate because project 0a9bff518ea94c908dc4903145b0841b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:04.358715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:04.359417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-25a31519-00f0-445e-9926-d4076321e231 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:04.360978 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1344/2682] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:04 2022] DELETE /identity/v3/projects/0a9bff518ea94c908dc4903145b0841b => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:04.366351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eac023-f0d6-4157-9934-a1ab620b6105 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.366842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eac023-f0d6-4157-9934-a1ab620b6105 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.366952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eac023-f0d6-4157-9934-a1ab620b6105 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.387857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-d4eac023-f0d6-4157-9934-a1ab620b6105 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:04.389060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d4eac023-f0d6-4157-9934-a1ab620b6105 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:04.455479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1339/2683] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:04.464674 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f190d954-349a-4dcf-977b-c1d9c360c686 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.464960 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f190d954-349a-4dcf-977b-c1d9c360c686 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.465233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f190d954-349a-4dcf-977b-c1d9c360c686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.485879 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f190d954-349a-4dcf-977b-c1d9c360c686 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:04.487050 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f190d954-349a-4dcf-977b-c1d9c360c686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:04.496434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1345/2684] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:04.500689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eac023-f0d6-4157-9934-a1ab620b6105 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.510690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eac023-f0d6-4157-9934-a1ab620b6105 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.515819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.516253 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.516534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.516935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.517591 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:04.518074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:04.520187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:04.532938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1340/2685] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:44:04.537911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f190d954-349a-4dcf-977b-c1d9c360c686 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.549113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f190d954-349a-4dcf-977b-c1d9c360c686 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.554528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.555160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.555440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.555710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.556337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:04.558217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:04.590115 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1346/2686] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:04 2022] POST /identity/v3/projects => generated 409 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:04.591319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.601762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e90f60-675d-48ae-b977-0f547490dba4 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.624537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:04.669911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.670523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.670796 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.671120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.671766 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:04.673695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:04.695915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1341/2687] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:04 2022] POST /identity/v3/users => generated 305 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:04.700275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.710543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4ec806-b99e-42ac-a553-8e1de21a00c6 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.715095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.715680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.715956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.716216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.716856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:04.718430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:04.728660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1347/2688] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:04.732470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.743359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b415ddd5-5c0b-4803-91ff-0c4330f1a5f0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.748094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1ac582-3903-4b4e-a086-a44943c8e182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.748672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1ac582-3903-4b4e-a086-a44943c8e182 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.748947 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1ac582-3903-4b4e-a086-a44943c8e182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.749215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1ac582-3903-4b4e-a086-a44943c8e182 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.749840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1ac582-3903-4b4e-a086-a44943c8e182 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:04.751450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1ac582-3903-4b4e-a086-a44943c8e182 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:04.761883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1342/2689] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:04.765905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.776865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6749eeb-9cd0-461e-8630-0932086a0725 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.781474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.782116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.782396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.782673 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] PATH_INFO: `/v3/projects/ced4d28d9c844bda92146ed26ee47836/users/c78b159070234f16b5240eff35f56491/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.803126 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ced4d28d9c844bda92146ed26ee47836, user_id=c78b159070234f16b5240eff35f56491, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:04.805819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:04.822188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1348/2690] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:04 2022] PUT /identity/v3/projects/ced4d28d9c844bda92146ed26ee47836/users/c78b159070234f16b5240eff35f56491/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:04.832912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69afb140-8a0f-4adc-bf8a-af7185c85366 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.833194 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69afb140-8a0f-4adc-bf8a-af7185c85366 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.833494 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69afb140-8a0f-4adc-bf8a-af7185c85366 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.862308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-69afb140-8a0f-4adc-bf8a-af7185c85366 None None] MFA Rules not processed for user `c78b159070234f16b5240eff35f56491`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:04.863560 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-69afb140-8a0f-4adc-bf8a-af7185c85366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:04.927106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1343/2691] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:04 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:04.931585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:04.991566 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63162a01-a929-4877-a903-5d6a942e5257 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:04.996416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:04.996942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:04.997217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:04.997486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:04.998113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.000042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.009123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:05.029915 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1349/2692] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:04 2022] POST /identity/v3/projects => generated 407 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:05.033982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69afb140-8a0f-4adc-bf8a-af7185c85366 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:05.044927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69afb140-8a0f-4adc-bf8a-af7185c85366 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:05.101132 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:05.101754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.102030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.102296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.102922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.104859 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.125678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1344/2693] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:05 2022] POST /identity/v3/users => generated 310 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:05.129805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:05.140238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50afa1d5-ef33-4618-af24-29f757812b81 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:05.145685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:05.146282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.146586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.146860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.147559 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.149163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.160547 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1350/2694] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:05.164949 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:05.175982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920d78cd-147c-42b5-966b-4c52c4901658 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:05.180655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:05.181308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.181583 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.181855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] PATH_INFO: `/v3/projects/a7bc2d41828a426d9118eed536baec23/users/a50d6b2c331e42f58581c474580a31a5/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.197175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7bc2d41828a426d9118eed536baec23, user_id=a50d6b2c331e42f58581c474580a31a5, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.199836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.208551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:05.215806 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1345/2695] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:05 2022] PUT /identity/v3/projects/a7bc2d41828a426d9118eed536baec23/users/a50d6b2c331e42f58581c474580a31a5/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:05.219735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:05.229663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47241f3-7dfb-4f67-a103-19bc14b51f9a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:05.286132 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff989e7-60b1-4635-aadf-8e62d729bf14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:05.286651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff989e7-60b1-4635-aadf-8e62d729bf14 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.286923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff989e7-60b1-4635-aadf-8e62d729bf14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.287278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff989e7-60b1-4635-aadf-8e62d729bf14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.287911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff989e7-60b1-4635-aadf-8e62d729bf14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.289503 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff989e7-60b1-4635-aadf-8e62d729bf14 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.299016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1351/2696] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:05 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:05.303484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:05.313771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44def88e-bb42-4df4-82cc-b116e45bb3da admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:05.319389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:05.320084 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.320366 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.320640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] PATH_INFO: `/v3/projects/a7bc2d41828a426d9118eed536baec23/users/a50d6b2c331e42f58581c474580a31a5/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.322570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7bc2d41828a426d9118eed536baec23, user_id=a50d6b2c331e42f58581c474580a31a5, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.325224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.336523 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1346/2697] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:05 2022] PUT /identity/v3/projects/a7bc2d41828a426d9118eed536baec23/users/a50d6b2c331e42f58581c474580a31a5/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:05.347371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.347680 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.347956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.379159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] MFA Rules not processed for user `a50d6b2c331e42f58581c474580a31a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:05.380402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:05.446323 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1352/2698] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:05 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:05.458181 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:05.459983 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:05.517153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a882a96-d46c-4f93-9284-216c42450019 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:05.521378 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:05.521951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.522217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.522529 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.531240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.532642 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.541866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1347/2699] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:05 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:05.701623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:05.703029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:05.714618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4206853f-2d27-43ca-a823-2c4af5462231 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:05.718142 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b14581-b478-42e8-8a22-f7dfa5e356c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:05.718727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b14581-b478-42e8-8a22-f7dfa5e356c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:05.719023 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b14581-b478-42e8-8a22-f7dfa5e356c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:05.719296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b14581-b478-42e8-8a22-f7dfa5e356c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:05.775412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b14581-b478-42e8-8a22-f7dfa5e356c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:05.776841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b14581-b478-42e8-8a22-f7dfa5e356c1 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:05.780403 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:05.788172 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1353/2700] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:05 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:15.537713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=32,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:15.788921 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:16.813077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:16.814485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:16.878895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3ec61-8e21-4843-8005-f30516e0d017 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:16.884037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:16.884654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:16.884931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:16.885200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] PATH_INFO: `/v3/users/8734279f63be407c892cbd8b9ec149c1` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:16.886309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8734279f63be407c892cbd8b9ec149c1)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:16.888221 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:16.932584 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8734279f63be407c892cbd8b9ec149c1'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:16.933214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8734279f63be407c892cbd8b9ec149c1'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:16.939940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8734279f63be407c892cbd8b9ec149c1'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:16.952043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8734279f63be407c892cbd8b9ec149c1'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:16.952656 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8734279f63be407c892cbd8b9ec149c1'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:16.965176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1348/2701] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:16 2022] DELETE /identity/v3/users/8734279f63be407c892cbd8b9ec149c1 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:16.969695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:16.970173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:16.970173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:16.989623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:16.991091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:17.049557 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1354/2702] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:17.059716 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:17.061100 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:17.123807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d1666a-9723-4bc8-847b-737d79384120 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:17.127877 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:17.128520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:17.128762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:17.129030 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:17.139240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:17.140866 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:17.152149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1349/2703] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:17.427758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:17.429218 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:17.444992 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16f47ec-80cc-4078-a320-b9e62411bf35 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:17.449950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:17.450590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:17.450901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:17.451159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] PATH_INFO: `/v3/projects/5626a34eb04e460b89f3d4a01d9d1c80` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:17.452342 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5626a34eb04e460b89f3d4a01d9d1c80)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:17.454261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:17.501784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5626a34eb04e460b89f3d4a01d9d1c80', 'request_id': 'req-a645ffba-0fd5-47c6-b868-dcf25554d91b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.509606 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5626a34eb04e460b89f3d4a01d9d1c80', 'request_id': 'req-a645ffba-0fd5-47c6-b868-dcf25554d91b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.516092 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5626a34eb04e460b89f3d4a01d9d1c80', 'request_id': 'req-a645ffba-0fd5-47c6-b868-dcf25554d91b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.528158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] The token cache is being invalidate because project 5626a34eb04e460b89f3d4a01d9d1c80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:17.528447 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.529078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.530777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1355/2704] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:17 2022] DELETE /identity/v3/projects/5626a34eb04e460b89f3d4a01d9d1c80 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:17.534525 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:17.544794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89deddd3-69c3-4046-8071-158a0151fafa service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:17.563597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:17.603589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:17.604205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:17.604484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:17.604759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] PATH_INFO: `/v3/users/502bcb154c28416c9e1523fee0739f38` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:17.605861 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] RBAC: Authorizing `identity:delete_user(user_id=502bcb154c28416c9e1523fee0739f38)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:17.607788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:17.654834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '502bcb154c28416c9e1523fee0739f38'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.655502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '502bcb154c28416c9e1523fee0739f38'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.662534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '502bcb154c28416c9e1523fee0739f38'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.674558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '502bcb154c28416c9e1523fee0739f38'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.675251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-12d69545-f907-4f27-83ba-0c537618feef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '502bcb154c28416c9e1523fee0739f38'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:17.688186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1350/2705] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:17 2022] DELETE /identity/v3/users/502bcb154c28416c9e1523fee0739f38 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:17.955267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:17.956746 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:17.970226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a645ffba-0fd5-47c6-b868-dcf25554d91b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.038204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.038750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.039083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.039405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] PATH_INFO: `/v3/projects/cc87f5aac6254615a4b3bc27303b8407` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.040521 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc87f5aac6254615a4b3bc27303b8407)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.042442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.073425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:18.090650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc87f5aac6254615a4b3bc27303b8407', 'request_id': 'req-654b0252-03e4-4ebd-9d65-3f97a8800326'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.096011 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc87f5aac6254615a4b3bc27303b8407', 'request_id': 'req-654b0252-03e4-4ebd-9d65-3f97a8800326'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.102707 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc87f5aac6254615a4b3bc27303b8407', 'request_id': 'req-654b0252-03e4-4ebd-9d65-3f97a8800326'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.115535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] The token cache is being invalidate because project cc87f5aac6254615a4b3bc27303b8407 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:18.115820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.116441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-654b0252-03e4-4ebd-9d65-3f97a8800326 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.117929 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1356/2706] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:17 2022] DELETE /identity/v3/projects/cc87f5aac6254615a4b3bc27303b8407 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:18.123011 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feec0529-c21c-4bd5-9fba-d5081c122b21 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.123485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feec0529-c21c-4bd5-9fba-d5081c122b21 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.123611 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feec0529-c21c-4bd5-9fba-d5081c122b21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.142672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-feec0529-c21c-4bd5-9fba-d5081c122b21 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:18.143901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-feec0529-c21c-4bd5-9fba-d5081c122b21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:18.203978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1351/2707] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:18.212864 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23ce8d-a9bf-46d9-bcec-a08cebcce0a6 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.213149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23ce8d-a9bf-46d9-bcec-a08cebcce0a6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.213424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23ce8d-a9bf-46d9-bcec-a08cebcce0a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.235024 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-6c23ce8d-a9bf-46d9-bcec-a08cebcce0a6 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:18.237785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6c23ce8d-a9bf-46d9-bcec-a08cebcce0a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:18.250074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1357/2708] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:18.252164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feec0529-c21c-4bd5-9fba-d5081c122b21 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.262026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feec0529-c21c-4bd5-9fba-d5081c122b21 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.266698 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.267325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.267600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.267997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.268522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:18.268821 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.270920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.283174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1352/2709] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:44:18.288484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23ce8d-a9bf-46d9-bcec-a08cebcce0a6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.298373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23ce8d-a9bf-46d9-bcec-a08cebcce0a6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.303303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.303883 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.304159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.304430 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.305119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.307030 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.336043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1358/2710] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/projects => generated 355 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:18.340174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.350455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4f7d88-ef94-4370-90d2-8d0e330fcb81 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.376054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.377741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:18.379543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:18.412849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.413451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.413729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.413997 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.414630 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.416614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.442867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1353/2711] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/users => generated 278 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:18.444056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.445987 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f87bba7-89ad-4207-8a8b-21470a621c61 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.450698 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.451356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.451637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.451912 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] PATH_INFO: `/v3/users/c78b159070234f16b5240eff35f56491` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.453049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c78b159070234f16b5240eff35f56491)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.453516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432d2ae1-b78e-4ea7-81da-e1337ae43e52 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.454997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.458121 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.458715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.458994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.459270 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.459900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.461461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.475008 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1354/2712] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:18 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:18.477749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.488722 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb445aa-bab7-43b0-a94c-49351c0efc11 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.494775 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.495697 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.496016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.496238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.496883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.498590 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.504311 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c78b159070234f16b5240eff35f56491'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.504984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c78b159070234f16b5240eff35f56491'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.509175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1355/2713] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:18.512840 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.514744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c78b159070234f16b5240eff35f56491'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.515483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c78b159070234f16b5240eff35f56491'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.522742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ce0e7e-574d-4a8d-b9af-daa97812df1c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.527483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96025d45-5d6e-4ac3-ad08-053d4df570c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.528096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96025d45-5d6e-4ac3-ad08-053d4df570c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.528370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96025d45-5d6e-4ac3-ad08-053d4df570c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.528533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-de9e600a-7a3d-4f57-8b95-f4e732853669 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c78b159070234f16b5240eff35f56491'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:18.528649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96025d45-5d6e-4ac3-ad08-053d4df570c3 admin admin] PATH_INFO: `/v3/projects/99296d28b25f4b25ab36ce58d7950bad/users/c01c00edecf747d5972d3267b0af0e46/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.541417 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1359/2714] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:18 2022] DELETE /identity/v3/users/c78b159070234f16b5240eff35f56491 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:18.543743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96025d45-5d6e-4ac3-ad08-053d4df570c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=99296d28b25f4b25ab36ce58d7950bad, user_id=c01c00edecf747d5972d3267b0af0e46, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.545678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.545966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.546298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.546425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96025d45-5d6e-4ac3-ad08-053d4df570c3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.554784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=16,keystone:SELECT=81,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:18.560238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1356/2715] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:18 2022] PUT /identity/v3/projects/99296d28b25f4b25ab36ce58d7950bad/users/c01c00edecf747d5972d3267b0af0e46/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:18.568316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:18.570400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:18.571589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.571890 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.572156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.600165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] MFA Rules not processed for user `c01c00edecf747d5972d3267b0af0e46`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:18.601389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:18.671447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1357/2716] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:18.675749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.677188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:18.747182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab7e891-8c20-4dae-9366-997c78602633 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.751932 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.752747 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.753022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.753319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.753577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1360/2717] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:44:18.753941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.755857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.762110 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:18.765436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.766873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:18.793887 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1358/2718] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/projects => generated 355 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:18.795479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.829789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe56687e-9f5e-45ee-a1b4-09ebac535856 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.860912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fb7b1-5fa5-4b71-892b-59f0c68afef6 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.865571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.866161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.866434 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.866700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.867355 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.869235 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.883977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.884667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.884925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.885204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.892300 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1359/2719] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:18 2022] POST /identity/v3/users => generated 284 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:18.896105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.898405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.899877 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.907829 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91e819-b44c-4d73-a760-498c35c58fe6 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.911797 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1361/2720] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:44:18.913458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.914215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.914596 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.914933 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.915666 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.917429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.931345 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1360/2721] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:44:18.932459 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:18.942079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abbe8fd-5614-428c-a6af-1e565f3cb538 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:18.947830 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:18.948642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:18.949061 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:18.949328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] PATH_INFO: `/v3/projects/654db61aee1f490ba2f71b790ba0b58d/users/8758ffa10e8a4aee8dfdc7eab0f3199b/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:18.965886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=654db61aee1f490ba2f71b790ba0b58d, user_id=8758ffa10e8a4aee8dfdc7eab0f3199b, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:18.968586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:18.984803 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1362/2722] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:18 2022] PUT /identity/v3/projects/654db61aee1f490ba2f71b790ba0b58d/users/8758ffa10e8a4aee8dfdc7eab0f3199b/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:18.988870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:19.001556 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182f97cb-7c07-4bd8-9394-a248205a3bb5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:19.061293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87667d7d-7728-4684-a94c-25bffe2e3aa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:19.061905 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87667d7d-7728-4684-a94c-25bffe2e3aa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.062274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87667d7d-7728-4684-a94c-25bffe2e3aa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.063149 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87667d7d-7728-4684-a94c-25bffe2e3aa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.063284 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87667d7d-7728-4684-a94c-25bffe2e3aa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:19.064868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87667d7d-7728-4684-a94c-25bffe2e3aa7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:19.075817 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1361/2723] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:18 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:19.079710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:19.091171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee390a-7e9f-4a7d-9b80-e767ee9bb8ca admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:19.096510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:19.097163 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.097438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.097713 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] PATH_INFO: `/v3/projects/654db61aee1f490ba2f71b790ba0b58d/users/8758ffa10e8a4aee8dfdc7eab0f3199b/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.099631 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=654db61aee1f490ba2f71b790ba0b58d, user_id=8758ffa10e8a4aee8dfdc7eab0f3199b, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:19.103043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:19.115070 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1363/2724] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:19 2022] PUT /identity/v3/projects/654db61aee1f490ba2f71b790ba0b58d/users/8758ffa10e8a4aee8dfdc7eab0f3199b/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:19.126057 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.126622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.126739 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.157186 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] MFA Rules not processed for user `8758ffa10e8a4aee8dfdc7eab0f3199b`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:19.158416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:19.203472 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:19.205284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:19.218910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:19.250772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1362/2725] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:19 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:44:19.262420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:19.264283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:19.267407 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:19.305440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac77867-0bf9-456d-8268-ad389289bbb6 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:19.310798 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:19.311458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.311710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.312255 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] PATH_INFO: `/v3/projects/ced4d28d9c844bda92146ed26ee47836` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.313269 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ced4d28d9c844bda92146ed26ee47836)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:19.315249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:19.328033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5434001-0642-4b7f-be63-f7b32448422e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:19.332321 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:19.332917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.333197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.333672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.343901 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:19.346497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:19.359794 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1363/2726] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:19 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:44:19.368134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ced4d28d9c844bda92146ed26ee47836', 'request_id': 'req-72a70bad-6116-4ba2-81b1-dec9bd92db59'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.374022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ced4d28d9c844bda92146ed26ee47836', 'request_id': 'req-72a70bad-6116-4ba2-81b1-dec9bd92db59'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.380867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ced4d28d9c844bda92146ed26ee47836', 'request_id': 'req-72a70bad-6116-4ba2-81b1-dec9bd92db59'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.393567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] The token cache is being invalidate because project ced4d28d9c844bda92146ed26ee47836 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:19.393946 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.394647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.396238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1364/2727] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:19 2022] DELETE /identity/v3/projects/ced4d28d9c844bda92146ed26ee47836 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:19.400670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:19.414207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f22c0cd-bc50-4df8-9569-9732a328fe75 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:19.446762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:19.448291 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:19.476924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:19.477637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.477992 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.478281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] PATH_INFO: `/v3/users/a50d6b2c331e42f58581c474580a31a5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.479445 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a50d6b2c331e42f58581c474580a31a5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:19.481348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:19.509592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a70bad-6116-4ba2-81b1-dec9bd92db59 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:19.513355 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351eb52e-8f9d-484b-9244-7044fe8ba8c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:19.514000 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351eb52e-8f9d-484b-9244-7044fe8ba8c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.514281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351eb52e-8f9d-484b-9244-7044fe8ba8c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.514536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351eb52e-8f9d-484b-9244-7044fe8ba8c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.516552 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-351eb52e-8f9d-484b-9244-7044fe8ba8c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:19.532552 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a50d6b2c331e42f58581c474580a31a5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.533352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a50d6b2c331e42f58581c474580a31a5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.541396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a50d6b2c331e42f58581c474580a31a5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.554513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a50d6b2c331e42f58581c474580a31a5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.555252 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a50d6b2c331e42f58581c474580a31a5'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.570171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1364/2728] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:19 2022] DELETE /identity/v3/users/a50d6b2c331e42f58581c474580a31a5 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:19.582827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351eb52e-8f9d-484b-9244-7044fe8ba8c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:19.584383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351eb52e-8f9d-484b-9244-7044fe8ba8c3 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:19.594633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1365/2729] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:19 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:44:19.861819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:19.863317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:19.875500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:19.932819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7639048-55db-4a90-818d-b5da7a22a478 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:19.937320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:19.937939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:19.938220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:19.938491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] PATH_INFO: `/v3/projects/a7bc2d41828a426d9118eed536baec23` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:19.939617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7bc2d41828a426d9118eed536baec23)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:19.941493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:19.986846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7bc2d41828a426d9118eed536baec23', 'request_id': 'req-e8528fd5-e847-4c72-9302-60603719b3b7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.992996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7bc2d41828a426d9118eed536baec23', 'request_id': 'req-e8528fd5-e847-4c72-9302-60603719b3b7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:19.998641 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7bc2d41828a426d9118eed536baec23', 'request_id': 'req-e8528fd5-e847-4c72-9302-60603719b3b7'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:20.010484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] The token cache is being invalidate because project a7bc2d41828a426d9118eed536baec23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:20.010765 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:20.011391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-e8528fd5-e847-4c72-9302-60603719b3b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:20.012842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1365/2730] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:19 2022] DELETE /identity/v3/projects/a7bc2d41828a426d9118eed536baec23 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:20.018184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf3a841-73a0-4a51-bbf7-8d5fb7644f2f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.018811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf3a841-73a0-4a51-bbf7-8d5fb7644f2f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.019007 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf3a841-73a0-4a51-bbf7-8d5fb7644f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.039577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ccf3a841-73a0-4a51-bbf7-8d5fb7644f2f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:20.040756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ccf3a841-73a0-4a51-bbf7-8d5fb7644f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:20.045920 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:20.106865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1366/2731] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:20.114598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418621d-b21f-4675-b54f-46ac03780923 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.114881 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418621d-b21f-4675-b54f-46ac03780923 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.115178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7418621d-b21f-4675-b54f-46ac03780923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.135480 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7418621d-b21f-4675-b54f-46ac03780923 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:20.136686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7418621d-b21f-4675-b54f-46ac03780923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:20.144535 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1366/2732] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:20.148852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf3a841-73a0-4a51-bbf7-8d5fb7644f2f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.158320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf3a841-73a0-4a51-bbf7-8d5fb7644f2f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.163524 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.164119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.164394 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.164662 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.165296 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:20.165589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.167196 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.179019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1367/2733] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:44:20.183103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418621d-b21f-4675-b54f-46ac03780923 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.192298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7418621d-b21f-4675-b54f-46ac03780923 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.196912 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.197484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.197757 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.198026 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.198651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.200633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.228379 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1367/2734] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:20 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:20.232515 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.243035 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b50d266-6965-4ebe-abdc-ee62a2572943 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.298593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.299216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.299494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.299765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.300395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.302295 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.325192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1368/2735] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:20 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:20.329280 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.341273 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b75d648-3539-49cd-a797-df39a46c2c30 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.345867 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.346455 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.346736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.347045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.347687 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.349269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.360047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1368/2736] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:20.363941 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.375884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8682c225-c3cb-489c-8a28-07a47baeef41 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.380873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f216d0-cee3-402d-a916-e4ed351a1dd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.381484 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f216d0-cee3-402d-a916-e4ed351a1dd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.381764 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f216d0-cee3-402d-a916-e4ed351a1dd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.382192 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f216d0-cee3-402d-a916-e4ed351a1dd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.382836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f216d0-cee3-402d-a916-e4ed351a1dd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.384654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f216d0-cee3-402d-a916-e4ed351a1dd4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.395679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1369/2737] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:44:20.400275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.413166 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfbc6d3-5219-42f6-b628-c98deb969ee2 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.418111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.418763 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.419075 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.419350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] PATH_INFO: `/v3/projects/292d31e332264cb3a5aa60b412592957/users/52e9b1a81eea497290aa43ec204512c2/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.436809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] RBAC: Authorizing `identity:create_grant(project_id=292d31e332264cb3a5aa60b412592957, user_id=52e9b1a81eea497290aa43ec204512c2, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.438438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.453304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1369/2738] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:20 2022] PUT /identity/v3/projects/292d31e332264cb3a5aa60b412592957/users/52e9b1a81eea497290aa43ec204512c2/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:20.463585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96748dfa-2852-412a-9db8-a418e4a1d60d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.463872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96748dfa-2852-412a-9db8-a418e4a1d60d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.464180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96748dfa-2852-412a-9db8-a418e4a1d60d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.492957 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-96748dfa-2852-412a-9db8-a418e4a1d60d None None] MFA Rules not processed for user `52e9b1a81eea497290aa43ec204512c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:20.494190 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-96748dfa-2852-412a-9db8-a418e4a1d60d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:20.556709 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1370/2739] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:20.561309 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.599668 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:20.624825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c33e62-374d-424f-ad2a-c86b4ad85280 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.629354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.629941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.630215 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.630482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.631202 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.633200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.659035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1370/2740] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:20 2022] POST /identity/v3/projects => generated 385 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:20.663187 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96748dfa-2852-412a-9db8-a418e4a1d60d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.673670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96748dfa-2852-412a-9db8-a418e4a1d60d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.677104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:20.735480 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.736090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.736363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.736629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.737262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.739193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.763539 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1371/2741] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:20 2022] POST /identity/v3/users => generated 299 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:20.767619 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.777761 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7491ebb9-3efc-4327-b064-310315e56c8d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.782738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.783191 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.783470 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.783841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.784358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.785971 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.796951 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1371/2742] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:20.800956 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.811835 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a3224e-b352-4d6a-b2ec-142940518169 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.817243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.818021 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.818302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.818592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] PATH_INFO: `/v3/projects/bb8776238a5b4f40bbe8e2a79912cc1e/users/a30c9812cad64066a00ee03734345e41/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.834012 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb8776238a5b4f40bbe8e2a79912cc1e, user_id=a30c9812cad64066a00ee03734345e41, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.837327 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.852443 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1372/2743] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:20 2022] PUT /identity/v3/projects/bb8776238a5b4f40bbe8e2a79912cc1e/users/a30c9812cad64066a00ee03734345e41/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:20.856731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.868768 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9f7331-559e-431e-a36a-e06b5b3ba641 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.940294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d05e8-a188-4dc6-9cc6-67d51da27832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.940974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d05e8-a188-4dc6-9cc6-67d51da27832 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.941250 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d05e8-a188-4dc6-9cc6-67d51da27832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.941518 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d05e8-a188-4dc6-9cc6-67d51da27832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.942148 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936d05e8-a188-4dc6-9cc6-67d51da27832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.943847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936d05e8-a188-4dc6-9cc6-67d51da27832 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.953534 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1372/2744] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:20 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:20.958101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:20.968903 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914cb226-5159-49fb-b2f0-bcf97ea300ec admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:20.973544 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:20.974226 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:20.974505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:20.975477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] PATH_INFO: `/v3/projects/bb8776238a5b4f40bbe8e2a79912cc1e/users/a30c9812cad64066a00ee03734345e41/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:20.977714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb8776238a5b4f40bbe8e2a79912cc1e, user_id=a30c9812cad64066a00ee03734345e41, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:20.980963 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:20.992110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1373/2745] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:20 2022] PUT /identity/v3/projects/bb8776238a5b4f40bbe8e2a79912cc1e/users/a30c9812cad64066a00ee03734345e41/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:21.003324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:21.003613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:21.003916 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:21.032910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] MFA Rules not processed for user `a30c9812cad64066a00ee03734345e41`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:21.034114 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:21.102303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1373/2746] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:21 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:21.112741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:21.114448 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:21.172010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc64e963-e074-4fef-8206-4b87c9c1d186 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:21.175477 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:21.176085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:21.176341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:21.176610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:21.232095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:21.233562 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:21.242795 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1374/2747] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:23.934032 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:23.935566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:23.974007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:23.999156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72889d86-7b0f-421d-b90a-77fd5d9f2184 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:24.002731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:24.003358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:24.003631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:24.003899 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:24.005620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:24.023713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:24.025617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:24.035244 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1374/2748] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:24.500344 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:24.501841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:24.517264 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:24.575285 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd732ccf-25da-40ab-b4c0-9c142e1bd3c7 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:24.580167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:24.580770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:24.581048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:24.581320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:24.583093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:24.594456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:24.596055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:24.606534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1375/2749] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:26.667753 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:26.669260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:26.706936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5fd99a-d717-4fcc-b0c7-d879a2e00886 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:26.714784 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:26.715440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:26.715982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:26.717458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:26.741910 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:26.743633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:26.760883 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1375/2750] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:26 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:27.446043 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:27.447602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:27.526860 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a1731c-1107-45e3-9b5a-8a9fd23abe91 service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:27.532300 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c2e6b7-d1f8-4e9a-bb8e-dd02dd10b97d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:27.532918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2e6b7-d1f8-4e9a-bb8e-dd02dd10b97d service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:27.533191 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2e6b7-d1f8-4e9a-bb8e-dd02dd10b97d service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:27.533457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2e6b7-d1f8-4e9a-bb8e-dd02dd10b97d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:27.535294 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-88c2e6b7-d1f8-4e9a-bb8e-dd02dd10b97d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:27.548891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c2e6b7-d1f8-4e9a-bb8e-dd02dd10b97d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:27.550500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c2e6b7-d1f8-4e9a-bb8e-dd02dd10b97d service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:27.561836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1376/2751] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:31.722219 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:31.792942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c7101c-d106-4660-b869-2d721cc88156 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:31.796724 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:31.797352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:31.797629 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:31.798013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] PATH_INFO: `/v3/users/52e9b1a81eea497290aa43ec204512c2` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:31.799035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52e9b1a81eea497290aa43ec204512c2)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:31.800975 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:31.854036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52e9b1a81eea497290aa43ec204512c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:31.854678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52e9b1a81eea497290aa43ec204512c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:31.862207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52e9b1a81eea497290aa43ec204512c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:31.875880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52e9b1a81eea497290aa43ec204512c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:31.876613 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52e9b1a81eea497290aa43ec204512c2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:31.892036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1376/2752] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:31 2022] DELETE /identity/v3/users/52e9b1a81eea497290aa43ec204512c2 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:31.896508 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:31.897013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:31.897420 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:31.919847 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:31.921034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:31.982771 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1377/2753] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:44:31.993471 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:31.995502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:32.047176 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:32.065269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f04f6b-9d9f-47f0-9a47-e2c55fa48806 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:32.068941 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:32.070640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:32.070924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:32.071352 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:32.080358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:32.081772 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:32.091769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1377/2754] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:44:32.361040 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:32.362596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:32.375895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6224d7d-7df4-4190-8a83-566c37045010 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:32.380368 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:32.380991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:32.381263 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:32.381575 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] PATH_INFO: `/v3/projects/292d31e332264cb3a5aa60b412592957` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:32.382868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=292d31e332264cb3a5aa60b412592957)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:32.384821 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:32.409752 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:32.432787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '292d31e332264cb3a5aa60b412592957', 'request_id': 'req-7ab90194-705f-45f0-b865-293f0f5406bc'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.438214 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '292d31e332264cb3a5aa60b412592957', 'request_id': 'req-7ab90194-705f-45f0-b865-293f0f5406bc'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.445237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '292d31e332264cb3a5aa60b412592957', 'request_id': 'req-7ab90194-705f-45f0-b865-293f0f5406bc'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.457458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] The token cache is being invalidate because project 292d31e332264cb3a5aa60b412592957 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:32.457767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.458373 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.460650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1378/2755] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:32 2022] DELETE /identity/v3/projects/292d31e332264cb3a5aa60b412592957 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:32.464616 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:32.474303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad91fda-9c53-4889-878a-4d4fa635b062 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:32.532773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:32.533399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:32.533672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:32.533939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] PATH_INFO: `/v3/users/a30c9812cad64066a00ee03734345e41` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:32.535086 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a30c9812cad64066a00ee03734345e41)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:32.536979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:32.595377 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a30c9812cad64066a00ee03734345e41'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.597047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a30c9812cad64066a00ee03734345e41'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.607033 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a30c9812cad64066a00ee03734345e41'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.622220 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a30c9812cad64066a00ee03734345e41'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.622935 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-9b44e819-8ed4-4594-95cc-64fd6cc6982d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a30c9812cad64066a00ee03734345e41'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:32.643440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1378/2756] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:32 2022] DELETE /identity/v3/users/a30c9812cad64066a00ee03734345e41 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:33.023313 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.026026 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:33.043413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab90194-705f-45f0-b865-293f0f5406bc admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.103672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.104306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.104584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.104854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] PATH_INFO: `/v3/projects/bb8776238a5b4f40bbe8e2a79912cc1e` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.105975 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb8776238a5b4f40bbe8e2a79912cc1e)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.107901 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.155906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb8776238a5b4f40bbe8e2a79912cc1e', 'request_id': 'req-b0138d11-be32-44c7-a89a-26d64f370520'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:33.161379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb8776238a5b4f40bbe8e2a79912cc1e', 'request_id': 'req-b0138d11-be32-44c7-a89a-26d64f370520'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:33.168284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb8776238a5b4f40bbe8e2a79912cc1e', 'request_id': 'req-b0138d11-be32-44c7-a89a-26d64f370520'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:33.181693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] The token cache is being invalidate because project bb8776238a5b4f40bbe8e2a79912cc1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:33.181976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:33.182602 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-b0138d11-be32-44c7-a89a-26d64f370520 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:33.184065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1379/2757] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:33 2022] DELETE /identity/v3/projects/bb8776238a5b4f40bbe8e2a79912cc1e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:33.190023 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30e96f0-35e9-4e3f-b2c6-a68f2c7dfa58 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.190519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30e96f0-35e9-4e3f-b2c6-a68f2c7dfa58 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.190622 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30e96f0-35e9-4e3f-b2c6-a68f2c7dfa58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.212728 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-f30e96f0-35e9-4e3f-b2c6-a68f2c7dfa58 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:33.214038 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f30e96f0-35e9-4e3f-b2c6-a68f2c7dfa58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:33.288595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1379/2758] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:33.297173 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b789e1af-daca-4288-96df-9787c5c793c3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.297538 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b789e1af-daca-4288-96df-9787c5c793c3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.297904 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b789e1af-daca-4288-96df-9787c5c793c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.317525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b789e1af-daca-4288-96df-9787c5c793c3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:33.318720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b789e1af-daca-4288-96df-9787c5c793c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:33.326789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1380/2759] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:33.331122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30e96f0-35e9-4e3f-b2c6-a68f2c7dfa58 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.336748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:33.343346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30e96f0-35e9-4e3f-b2c6-a68f2c7dfa58 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.349281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.350054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.351062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.351449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.352198 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:33.352502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.354078 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.365882 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1380/2760] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:44:33.371034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b789e1af-daca-4288-96df-9787c5c793c3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.380193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b789e1af-daca-4288-96df-9787c5c793c3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.385505 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.386131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.386428 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.386705 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.387415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.389411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.419745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1381/2761] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:33 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:33.424163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.434969 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002d19eb-595d-469a-bc47-4aad1d96be97 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.503218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.503218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.503218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.503218 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.503621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.503621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.523493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1381/2762] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:33 2022] POST /identity/v3/users => generated 292 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:33.527489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.537482 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d851dcc9-b5e1-4608-987a-a866533b3046 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.542586 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.543215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.543488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.543755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.544386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.545952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.555604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1382/2763] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:33.559549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.570025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316c6e9e-a49a-4cbb-9fad-bf874156a056 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.574850 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d02f31-3141-40ed-aef9-d4bfc41f8d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.575475 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d02f31-3141-40ed-aef9-d4bfc41f8d97 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.575752 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d02f31-3141-40ed-aef9-d4bfc41f8d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.576022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d02f31-3141-40ed-aef9-d4bfc41f8d97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.576661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d02f31-3141-40ed-aef9-d4bfc41f8d97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.578242 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d02f31-3141-40ed-aef9-d4bfc41f8d97 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.589647 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1382/2764] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:33.593936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.599006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:33.605525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec6c478-aa6a-4c4f-bb03-a8b9a14aabb8 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.610889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.611554 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.611846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.612116 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] PATH_INFO: `/v3/projects/a7e3a58f8ec748588c130e24c46339df/users/ac60aae26a8a4df49f586f9e739bc158/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.626854 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7e3a58f8ec748588c130e24c46339df, user_id=ac60aae26a8a4df49f586f9e739bc158, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.629507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.642925 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1383/2765] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:33 2022] PUT /identity/v3/projects/a7e3a58f8ec748588c130e24c46339df/users/ac60aae26a8a4df49f586f9e739bc158/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:33.655643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7f9fff-a750-4398-8275-49cb2a360c17 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.655929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7f9fff-a750-4398-8275-49cb2a360c17 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.656203 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7f9fff-a750-4398-8275-49cb2a360c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.685170 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ac7f9fff-a750-4398-8275-49cb2a360c17 None None] MFA Rules not processed for user `ac60aae26a8a4df49f586f9e739bc158`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:33.686412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ac7f9fff-a750-4398-8275-49cb2a360c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:33.754204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1383/2766] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:33.766090 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.767809 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:33.825413 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08bef82-c44a-43d0-91d0-0a98d941638c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.828981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55820f90-3232-46c7-867e-05b1d3a530d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.829585 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55820f90-3232-46c7-867e-05b1d3a530d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.829862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55820f90-3232-46c7-867e-05b1d3a530d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.830131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55820f90-3232-46c7-867e-05b1d3a530d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.839944 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55820f90-3232-46c7-867e-05b1d3a530d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.841390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55820f90-3232-46c7-867e-05b1d3a530d1 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:33.851161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1384/2767] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:33.871678 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7f9fff-a750-4398-8275-49cb2a360c17 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:33.935953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:33.941208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7f9fff-a750-4398-8275-49cb2a360c17 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:33.948060 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:33.948860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:33.949322 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:33.949727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] PATH_INFO: `/v3/users/ac60aae26a8a4df49f586f9e739bc158` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:33.951426 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac60aae26a8a4df49f586f9e739bc158)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:33.954413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:34.011451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac60aae26a8a4df49f586f9e739bc158'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.012196 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac60aae26a8a4df49f586f9e739bc158'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.019715 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac60aae26a8a4df49f586f9e739bc158'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.033237 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac60aae26a8a4df49f586f9e739bc158'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.033956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac60aae26a8a4df49f586f9e739bc158'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.049520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1384/2768] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:33 2022] DELETE /identity/v3/users/ac60aae26a8a4df49f586f9e739bc158 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:34.054510 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:34.055199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:34.055199 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:34.074889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:34.076185 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:34.146695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1385/2769] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:34.156868 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:34.158269 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:34.338522 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6d79f1-b136-4c6e-8613-8c54a1e27412 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:34.342283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c21f68-268f-4a45-810a-e77ea4796e13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:34.342741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c21f68-268f-4a45-810a-e77ea4796e13 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:34.343054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c21f68-268f-4a45-810a-e77ea4796e13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:34.343325 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c21f68-268f-4a45-810a-e77ea4796e13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:34.352848 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c21f68-268f-4a45-810a-e77ea4796e13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:34.354298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c21f68-268f-4a45-810a-e77ea4796e13 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:34.363814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1385/2770] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:34.645671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:34.647105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:34.661923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba08a8c5-54fb-4a45-a602-e9127164682a None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:34.667865 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:34.673660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:34.674861 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:34.675933 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] PATH_INFO: `/v3/projects/a7e3a58f8ec748588c130e24c46339df` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:34.679338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7e3a58f8ec748588c130e24c46339df)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:34.683361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:34.743584 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7e3a58f8ec748588c130e24c46339df', 'request_id': 'req-c98a6d89-5e48-460c-9842-c71faa9cf86f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.753899 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7e3a58f8ec748588c130e24c46339df', 'request_id': 'req-c98a6d89-5e48-460c-9842-c71faa9cf86f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.760440 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7e3a58f8ec748588c130e24c46339df', 'request_id': 'req-c98a6d89-5e48-460c-9842-c71faa9cf86f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.773672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] The token cache is being invalidate because project a7e3a58f8ec748588c130e24c46339df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:34.773953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.774596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.776113 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1386/2771] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:34 2022] DELETE /identity/v3/projects/a7e3a58f8ec748588c130e24c46339df => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:34.781012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a96a5-8b96-446e-9ea5-1875bbf7d295 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:34.781294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a96a5-8b96-446e-9ea5-1875bbf7d295 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:34.781566 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a96a5-8b96-446e-9ea5-1875bbf7d295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:34.801000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ef4a96a5-8b96-446e-9ea5-1875bbf7d295 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:34.802205 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ef4a96a5-8b96-446e-9ea5-1875bbf7d295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:34.822085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:34.823512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:34.829616 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:34.863447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1386/2772] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:34.899346 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98a6d89-5e48-460c-9842-c71faa9cf86f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:34.906028 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:34.906685 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:34.906990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:34.907262 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] PATH_INFO: `/v3/users/c01c00edecf747d5972d3267b0af0e46` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:34.908471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c01c00edecf747d5972d3267b0af0e46)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:34.910432 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:34.929090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f9f2dc-6a0f-494b-867b-695913b9dfbf None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:34.929367 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f9f2dc-6a0f-494b-867b-695913b9dfbf None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:34.929632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f9f2dc-6a0f-494b-867b-695913b9dfbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:34.952561 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-63f9f2dc-6a0f-494b-867b-695913b9dfbf None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:34.954665 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-63f9f2dc-6a0f-494b-867b-695913b9dfbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:34.964977 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1387/2773] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:44:34.965950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c01c00edecf747d5972d3267b0af0e46'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.966804 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c01c00edecf747d5972d3267b0af0e46'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.970131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f9f2dc-6a0f-494b-867b-695913b9dfbf None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:34.975131 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c01c00edecf747d5972d3267b0af0e46'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.976341 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c01c00edecf747d5972d3267b0af0e46'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.980095 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f9f2dc-6a0f-494b-867b-695913b9dfbf None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:34.984812 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:34.985382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:34.985653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:34.985920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:34.986540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:34.986836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:34.988305 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-634b79f6-22fa-4d39-b898-8d37eaee9e21 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c01c00edecf747d5972d3267b0af0e46'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:34.988468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:34.998487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:35.001744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1387/2774] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:34 2022] DELETE /identity/v3/users/c01c00edecf747d5972d3267b0af0e46 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:35.001898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1388/2775] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:44:35.005810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.006089 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.006596 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.007725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.009878 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:35.023168 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876a00f-9490-4853-b6f9-c795b14ad205 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.027071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:35.028441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:35.090048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.090643 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.090915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.091209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.091879 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.093875 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.096545 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1388/2776] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:35.106390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.107808 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:35.129437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1389/2777] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:35 2022] POST /identity/v3/projects => generated 377 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:35.134482 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.140871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:35.180164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d2ced-0670-44b8-903a-2afbdac66070 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.203264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbae3ac-5d27-463c-bef0-b964b5c4a779 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.208598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.215477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.215477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.215477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.215477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.215477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.248952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1390/2778] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:35 2022] POST /identity/v3/users => generated 289 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:35.250334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.251409 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.252088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.252363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.252627 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.261695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070f4871-fe1a-4990-8530-a047a6583eec admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.262512 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.263964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.266824 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.268096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.269276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.269276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.269493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.271260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.274336 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1389/2779] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:44:35.284037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1391/2780] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:35.288335 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.309976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767e03d2-84d3-4fe3-ae36-b64cf8dd1f4b service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.315361 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c98a600-048e-405f-b316-427d7bb4653a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.316056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c98a600-048e-405f-b316-427d7bb4653a admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.316624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c98a600-048e-405f-b316-427d7bb4653a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.316897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c98a600-048e-405f-b316-427d7bb4653a admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.317543 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c98a600-048e-405f-b316-427d7bb4653a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.319714 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c98a600-048e-405f-b316-427d7bb4653a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.331643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1390/2781] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:35 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:44:35.336442 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.348391 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97117b55-4f85-4e1b-9f1c-5da7aa008f4d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.354410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.355216 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.355604 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.356004 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] PATH_INFO: `/v3/projects/e93f474e93ce417f9a81c5c496687a71/users/fba90caacf3e4a739f854e3ac0b090fd/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.371669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=e93f474e93ce417f9a81c5c496687a71, user_id=fba90caacf3e4a739f854e3ac0b090fd, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.374206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.388421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1392/2782] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:35 2022] PUT /identity/v3/projects/e93f474e93ce417f9a81c5c496687a71/users/fba90caacf3e4a739f854e3ac0b090fd/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:35.400317 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.400711 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.401048 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.433364 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] MFA Rules not processed for user `fba90caacf3e4a739f854e3ac0b090fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:35.434826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:35.503091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1391/2783] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:35 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:35.515118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.516836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:35.560893 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.563124 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:35.563526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:35.589446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b263ad80-df70-4d45-b447-e7f076c650ad admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.593473 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.594488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.594773 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.595081 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.604540 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.605965 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.615130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1393/2784] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:35 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:35.636800 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f4204b-3ed3-4a5f-ad1b-b8692863cd4c None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.641328 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.641984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.642261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.642531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] PATH_INFO: `/v3/projects/99296d28b25f4b25ab36ce58d7950bad` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.643642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=99296d28b25f4b25ab36ce58d7950bad)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.645532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.648494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:35.697231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '99296d28b25f4b25ab36ce58d7950bad', 'request_id': 'req-3d599537-b5fe-4089-a50f-4c54473433f4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.702922 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '99296d28b25f4b25ab36ce58d7950bad', 'request_id': 'req-3d599537-b5fe-4089-a50f-4c54473433f4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.711347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '99296d28b25f4b25ab36ce58d7950bad', 'request_id': 'req-3d599537-b5fe-4089-a50f-4c54473433f4'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.724215 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] The token cache is being invalidate because project 99296d28b25f4b25ab36ce58d7950bad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:35.724502 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.725158 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.727790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1392/2785] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:35 2022] DELETE /identity/v3/projects/99296d28b25f4b25ab36ce58d7950bad => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:35.731640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:35.744895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72faf35-d74a-4462-a9cc-2947b2d74c97 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:35.811305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:35.812108 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:35.812457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:35.812800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] PATH_INFO: `/v3/users/8758ffa10e8a4aee8dfdc7eab0f3199b` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:35.814089 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8758ffa10e8a4aee8dfdc7eab0f3199b)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:35.816111 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:35.867151 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8758ffa10e8a4aee8dfdc7eab0f3199b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.867826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8758ffa10e8a4aee8dfdc7eab0f3199b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.876614 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8758ffa10e8a4aee8dfdc7eab0f3199b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.890444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8758ffa10e8a4aee8dfdc7eab0f3199b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.891447 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-3ba77568-8527-4b0b-932e-0d785e1f0272 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8758ffa10e8a4aee8dfdc7eab0f3199b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:35.909441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1394/2786] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:35 2022] DELETE /identity/v3/users/8758ffa10e8a4aee8dfdc7eab0f3199b => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:36.211984 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.213397 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:36.228741 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d599537-b5fe-4089-a50f-4c54473433f4 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.291468 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.292093 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.292371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.292645 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] PATH_INFO: `/v3/projects/654db61aee1f490ba2f71b790ba0b58d` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.293759 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=654db61aee1f490ba2f71b790ba0b58d)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:36.295723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:36.345139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '654db61aee1f490ba2f71b790ba0b58d', 'request_id': 'req-50745471-9972-43c8-966b-1b141607da9f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:36.351451 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '654db61aee1f490ba2f71b790ba0b58d', 'request_id': 'req-50745471-9972-43c8-966b-1b141607da9f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:36.358273 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '654db61aee1f490ba2f71b790ba0b58d', 'request_id': 'req-50745471-9972-43c8-966b-1b141607da9f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:36.371843 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] The token cache is being invalidate because project 654db61aee1f490ba2f71b790ba0b58d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:36.372133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:36.372878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-50745471-9972-43c8-966b-1b141607da9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:36.374284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1393/2787] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:36 2022] DELETE /identity/v3/projects/654db61aee1f490ba2f71b790ba0b58d => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:36.379129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3647ed5e-1803-4b35-8c78-05f52ce767e5 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.379609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3647ed5e-1803-4b35-8c78-05f52ce767e5 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.379609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3647ed5e-1803-4b35-8c78-05f52ce767e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.400208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-3647ed5e-1803-4b35-8c78-05f52ce767e5 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:36.401437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-3647ed5e-1803-4b35-8c78-05f52ce767e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:36.463396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1395/2788] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:36.471314 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5583a5d6-f4f0-49d6-8b7f-6535bfb3b8f1 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.472022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5583a5d6-f4f0-49d6-8b7f-6535bfb3b8f1 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.472532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5583a5d6-f4f0-49d6-8b7f-6535bfb3b8f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.493895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5583a5d6-f4f0-49d6-8b7f-6535bfb3b8f1 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:36.495384 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5583a5d6-f4f0-49d6-8b7f-6535bfb3b8f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:36.503340 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1394/2789] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:44:36.507954 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3647ed5e-1803-4b35-8c78-05f52ce767e5 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.512819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:36.520015 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3647ed5e-1803-4b35-8c78-05f52ce767e5 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.527413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.528097 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.528439 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.528735 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.529369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:36.529653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:36.531274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:36.545502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1396/2790] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:44:36.549936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5583a5d6-f4f0-49d6-8b7f-6535bfb3b8f1 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.559727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5583a5d6-f4f0-49d6-8b7f-6535bfb3b8f1 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.564351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.564942 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.565217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.565483 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.566108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:36.568046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:36.583010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:36.598827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1395/2791] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:36 2022] POST /identity/v3/projects => generated 363 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:36.603721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.613852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b59615-e9ee-497d-ba65-c65ffbbc8c63 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.672102 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.672637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.672917 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.673187 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.673823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:36.675745 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:36.702754 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1397/2792] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:36 2022] POST /identity/v3/users => generated 282 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:36.704102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.714926 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fede72f3-7736-44bb-a229-a2970b32cd3b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.720082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.720375 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.720684 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.720951 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.721587 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:36.723270 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:36.737136 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1396/2793] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:36.738362 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.749896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca72562-c08d-47e2-8a99-6d25015adfa7 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.755117 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9a5957-ebe5-487a-a2de-376eb83b0ab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.755915 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9a5957-ebe5-487a-a2de-376eb83b0ab8 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.756276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9a5957-ebe5-487a-a2de-376eb83b0ab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.757296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9a5957-ebe5-487a-a2de-376eb83b0ab8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.757953 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9a5957-ebe5-487a-a2de-376eb83b0ab8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:36.760154 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9a5957-ebe5-487a-a2de-376eb83b0ab8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:36.771248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1398/2794] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:36.776002 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.785832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f136abb-83df-405c-8f04-d17092f1ad20 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.790823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.791481 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.791753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.792019 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] PATH_INFO: `/v3/projects/c402e3e27ff246febaac13fb21cef572/users/a09eb4ec8ecf4e47b50afb175eec97e5/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.806773 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c402e3e27ff246febaac13fb21cef572, user_id=a09eb4ec8ecf4e47b50afb175eec97e5, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:36.809404 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:36.822785 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1397/2795] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:36 2022] PUT /identity/v3/projects/c402e3e27ff246febaac13fb21cef572/users/a09eb4ec8ecf4e47b50afb175eec97e5/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:36.833021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba69b21e-d008-4509-af17-d914cf3bd767 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.833304 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba69b21e-d008-4509-af17-d914cf3bd767 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.833575 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba69b21e-d008-4509-af17-d914cf3bd767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.860855 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-ba69b21e-d008-4509-af17-d914cf3bd767 None None] MFA Rules not processed for user `a09eb4ec8ecf4e47b50afb175eec97e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:36.862066 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ba69b21e-d008-4509-af17-d914cf3bd767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:36.925274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1399/2796] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:36 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:36.929828 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:36.992156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e2f113-783e-4b3e-aec9-f8fc6bff3099 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:36.997655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:36.998316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:36.998593 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:36.998869 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:36.999918 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.001457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.032204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1398/2797] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:36 2022] POST /identity/v3/projects => generated 365 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:37.036324 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba69b21e-d008-4509-af17-d914cf3bd767 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:37.047487 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba69b21e-d008-4509-af17-d914cf3bd767 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:37.104059 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:37.111513 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:37.112514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.112792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.113065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.113713 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.116192 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.141372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1400/2798] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:37 2022] POST /identity/v3/users => generated 289 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:37.145411 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:37.155985 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550fba1f-49f1-4a4a-b19b-ca1d9aa4f4d9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:37.161423 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:37.162095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.162907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.162907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.163678 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.165293 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.176636 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1399/2799] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:44:37.181385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:37.191135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380316f7-14bd-43db-ab6e-e090a766d555 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:37.196987 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:37.197633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.197943 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.198225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] PATH_INFO: `/v3/projects/fda6ea30d67340b78385cc027cb4e313/users/a881f1b4c12a4fd49f4f1dd99902d3ea/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.213979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fda6ea30d67340b78385cc027cb4e313, user_id=a881f1b4c12a4fd49f4f1dd99902d3ea, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.216343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.229393 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1401/2800] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:37 2022] PUT /identity/v3/projects/fda6ea30d67340b78385cc027cb4e313/users/a881f1b4c12a4fd49f4f1dd99902d3ea/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:37.233621 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:37.245079 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264c83f6-f692-42ce-9c5c-5f57c7df65b9 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:37.299648 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58192ec-c8e6-4ed1-8716-bc881139a9bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:37.300237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58192ec-c8e6-4ed1-8716-bc881139a9bd admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.300526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58192ec-c8e6-4ed1-8716-bc881139a9bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.300789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58192ec-c8e6-4ed1-8716-bc881139a9bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.301441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58192ec-c8e6-4ed1-8716-bc881139a9bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.303057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58192ec-c8e6-4ed1-8716-bc881139a9bd admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.313108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1400/2801] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:37 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:37.317429 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:37.328123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706e8e0b-4326-4f18-b591-3c03cc85b1f8 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:37.332655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:37.333288 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.333563 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.333834 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] PATH_INFO: `/v3/projects/fda6ea30d67340b78385cc027cb4e313/users/a881f1b4c12a4fd49f4f1dd99902d3ea/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.335939 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fda6ea30d67340b78385cc027cb4e313, user_id=a881f1b4c12a4fd49f4f1dd99902d3ea, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.338526 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.348598 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1402/2802] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:37 2022] PUT /identity/v3/projects/fda6ea30d67340b78385cc027cb4e313/users/a881f1b4c12a4fd49f4f1dd99902d3ea/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:37.360145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da326f-e1a8-4020-923a-c800e83fb24d None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.360435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da326f-e1a8-4020-923a-c800e83fb24d None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.360710 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da326f-e1a8-4020-923a-c800e83fb24d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.389134 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-92da326f-e1a8-4020-923a-c800e83fb24d None None] MFA Rules not processed for user `a881f1b4c12a4fd49f4f1dd99902d3ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:37.390339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-92da326f-e1a8-4020-923a-c800e83fb24d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:37.406989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:37.460806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1401/2803] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:37 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:37.472238 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:37.473927 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:37.536738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400bb42b-197f-4bd0-822b-b907e740b9e5 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:37.541099 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:37.541689 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.541959 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.542226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.551595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.553065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.563488 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1403/2804] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:37 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:37.640723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92da326f-e1a8-4020-923a-c800e83fb24d None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:37.650658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92da326f-e1a8-4020-923a-c800e83fb24d None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:37.654774 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:37.655402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:37.655679 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:37.655949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:37.714383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:37.715827 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:37.724716 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1402/2805] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:37 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:39.660660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:39.662206 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:39.741821 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27119a9c-c7a9-4120-a7b1-480a99ecdb55 service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:39.747695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04de5606-62ef-4a87-a1a9-6f190311df18 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:39.748293 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04de5606-62ef-4a87-a1a9-6f190311df18 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:39.748570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04de5606-62ef-4a87-a1a9-6f190311df18 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:39.748841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04de5606-62ef-4a87-a1a9-6f190311df18 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:39.750545 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-04de5606-62ef-4a87-a1a9-6f190311df18 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:39.763381 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04de5606-62ef-4a87-a1a9-6f190311df18 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:39.764863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04de5606-62ef-4a87-a1a9-6f190311df18 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:39.776005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1404/2806] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:41.498416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:41.559471 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14266f0a-4848-4f56-a95b-bc7ce2b76a7e service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:41.563979 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:41.564594 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:41.564871 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:41.565145 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] PATH_INFO: `/v3/users/a09eb4ec8ecf4e47b50afb175eec97e5` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:41.566247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a09eb4ec8ecf4e47b50afb175eec97e5)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:41.568203 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:41.614682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a09eb4ec8ecf4e47b50afb175eec97e5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:41.615353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a09eb4ec8ecf4e47b50afb175eec97e5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:41.621523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:41.623533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a09eb4ec8ecf4e47b50afb175eec97e5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:41.624438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a09eb4ec8ecf4e47b50afb175eec97e5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:41.636923 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a09eb4ec8ecf4e47b50afb175eec97e5'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:41.649652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1403/2807] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:41 2022] DELETE /identity/v3/users/a09eb4ec8ecf4e47b50afb175eec97e5 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:41.654129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:41.654683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:41.654683 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:41.674769 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:41.675979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:41.707570 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:41.737485 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1405/2808] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:41.747318 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:41.748721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:41.804623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cb47c7-6cad-41bb-aa67-885e77396e0d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:41.808249 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:41.808841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:41.809118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:41.809387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:41.818346 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:41.820310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:41.830268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1404/2809] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:42.090804 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:42.092339 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:42.106182 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366aca8c-7e42-4009-aa8a-1d5f5d7a96d0 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:42.112177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:42.112989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:42.113397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:42.113537 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] PATH_INFO: `/v3/projects/c402e3e27ff246febaac13fb21cef572` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:42.114667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c402e3e27ff246febaac13fb21cef572)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:42.117153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:42.162062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c402e3e27ff246febaac13fb21cef572', 'request_id': 'req-a44997e0-7735-4b74-8df3-50ec2834848c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.168595 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c402e3e27ff246febaac13fb21cef572', 'request_id': 'req-a44997e0-7735-4b74-8df3-50ec2834848c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.173962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c402e3e27ff246febaac13fb21cef572', 'request_id': 'req-a44997e0-7735-4b74-8df3-50ec2834848c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.186667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] The token cache is being invalidate because project c402e3e27ff246febaac13fb21cef572 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:42.186993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.187609 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.189074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1406/2810] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:42 2022] DELETE /identity/v3/projects/c402e3e27ff246febaac13fb21cef572 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:42.193265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:42.203629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb12873-b82e-4f7f-a4b5-e965db39f746 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:42.263343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:42.263981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:42.264257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:42.264528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] PATH_INFO: `/v3/users/a881f1b4c12a4fd49f4f1dd99902d3ea` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:42.265658 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=a881f1b4c12a4fd49f4f1dd99902d3ea)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:42.267600 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:42.314410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a881f1b4c12a4fd49f4f1dd99902d3ea'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.315071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a881f1b4c12a4fd49f4f1dd99902d3ea'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.322094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a881f1b4c12a4fd49f4f1dd99902d3ea'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.322726 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a881f1b4c12a4fd49f4f1dd99902d3ea'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.335509 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-a70236a1-ce3c-4663-8bbe-898d70ed31ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a881f1b4c12a4fd49f4f1dd99902d3ea'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.338225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:42.351303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1405/2811] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:42 2022] DELETE /identity/v3/users/a881f1b4c12a4fd49f4f1dd99902d3ea => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:42.623731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:42.625231 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:42.640076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44997e0-7735-4b74-8df3-50ec2834848c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:42.697623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:42.698251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:42.698512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:42.698788 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] PATH_INFO: `/v3/projects/fda6ea30d67340b78385cc027cb4e313` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:42.699919 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fda6ea30d67340b78385cc027cb4e313)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:42.701799 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:42.748224 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fda6ea30d67340b78385cc027cb4e313', 'request_id': 'req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.754243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fda6ea30d67340b78385cc027cb4e313', 'request_id': 'req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.761800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fda6ea30d67340b78385cc027cb4e313', 'request_id': 'req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.764135 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:42.777389 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] The token cache is being invalidate because project fda6ea30d67340b78385cc027cb4e313 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:42.777694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.778565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8a5b0ca4-d3e9-4c4b-b5f5-894c1fb73bd8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:42.780113 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1407/2812] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:42 2022] DELETE /identity/v3/projects/fda6ea30d67340b78385cc027cb4e313 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:42.784947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9752b284-7d83-426a-90c9-7968d8049668 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:42.785436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9752b284-7d83-426a-90c9-7968d8049668 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:42.785536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9752b284-7d83-426a-90c9-7968d8049668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:42.805789 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9752b284-7d83-426a-90c9-7968d8049668 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:42.807100 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9752b284-7d83-426a-90c9-7968d8049668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:42.866377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1406/2813] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:44:42.874431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1228b5-8ef5-48bb-83c4-3ffd750ce793 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:42.874651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1228b5-8ef5-48bb-83c4-3ffd750ce793 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:42.874920 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1228b5-8ef5-48bb-83c4-3ffd750ce793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:42.894826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-1f1228b5-8ef5-48bb-83c4-3ffd750ce793 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:42.896054 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1f1228b5-8ef5-48bb-83c4-3ffd750ce793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:42.905209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1408/2814] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:42.909232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9752b284-7d83-426a-90c9-7968d8049668 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:42.920110 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9752b284-7d83-426a-90c9-7968d8049668 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:42.924743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:42.925338 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:42.925617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:42.925907 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:42.926551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:42.926855 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:42.929609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:42.940947 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1407/2815] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:44:42.945041 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1228b5-8ef5-48bb-83c4-3ffd750ce793 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:42.954062 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1228b5-8ef5-48bb-83c4-3ffd750ce793 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:42.958554 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:42.959162 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:42.959994 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:42.960260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:42.960869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:42.962748 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:42.989188 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1409/2816] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:42 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:42.993751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.005104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4a88d2-25b5-4034-bae7-a0e8e090e961 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.062513 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.063148 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.063427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.063697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.064321 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.066232 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.087618 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1408/2817] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:42 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:43.091811 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.102701 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfd44ea-6ddb-4b1a-b977-9b1fb132fe60 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.107336 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.107918 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.108193 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.108461 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.109098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.110658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.120842 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1410/2818] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:43.124780 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.134888 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a988ad-b784-4368-87ad-14386a820247 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.139511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622f50ae-4cbb-44b3-a7d3-9a8281d86fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.140095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f50ae-4cbb-44b3-a7d3-9a8281d86fc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.140377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f50ae-4cbb-44b3-a7d3-9a8281d86fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.140647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622f50ae-4cbb-44b3-a7d3-9a8281d86fc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.141278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622f50ae-4cbb-44b3-a7d3-9a8281d86fc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.142859 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622f50ae-4cbb-44b3-a7d3-9a8281d86fc0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.153841 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1409/2819] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:43.157778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.168267 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e3609-1269-449f-8e61-7b340e31dcb0 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.172807 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.173437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.173709 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.173978 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] PATH_INFO: `/v3/projects/b1e5d4726c054a16b1f7c8e897432d8c/users/fcdf07e337cc470a89ad83025fba1a3b/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.188120 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1e5d4726c054a16b1f7c8e897432d8c, user_id=fcdf07e337cc470a89ad83025fba1a3b, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.190705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.203521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1411/2820] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:43 2022] PUT /identity/v3/projects/b1e5d4726c054a16b1f7c8e897432d8c/users/fcdf07e337cc470a89ad83025fba1a3b/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:43.214551 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cacc293-731e-4d64-8ac5-e6f41c1c5988 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.215059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cacc293-731e-4d64-8ac5-e6f41c1c5988 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.215348 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cacc293-731e-4d64-8ac5-e6f41c1c5988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.244147 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-0cacc293-731e-4d64-8ac5-e6f41c1c5988 None None] MFA Rules not processed for user `fcdf07e337cc470a89ad83025fba1a3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:43.245595 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-0cacc293-731e-4d64-8ac5-e6f41c1c5988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:43.291287 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:43.309455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1410/2821] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:43.313886 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.374299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1120a7-6078-42d8-8a24-4a80cfc381bc admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.378847 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.379396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.379670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.379934 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.380541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.383082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.414260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1412/2822] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:43 2022] POST /identity/v3/projects => generated 377 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:43.415268 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cacc293-731e-4d64-8ac5-e6f41c1c5988 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.425836 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cacc293-731e-4d64-8ac5-e6f41c1c5988 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.483592 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.484204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.484478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.484743 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.485378 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.487303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.510703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1411/2823] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:43 2022] POST /identity/v3/users => generated 289 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:43.514449 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.525147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0a2250-1ad4-4250-a5a3-9a1eadb42e70 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.536101 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.536702 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.536976 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.537239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.538402 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.540161 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.554453 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1413/2824] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:43 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:44:43.555671 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.565565 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb25b093-5bc9-4689-b033-0f3df23119df admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.570796 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc98e8-740c-45b5-af6f-0518ef8493d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.571425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc98e8-740c-45b5-af6f-0518ef8493d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.571717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc98e8-740c-45b5-af6f-0518ef8493d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.571976 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc98e8-740c-45b5-af6f-0518ef8493d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.572613 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dc98e8-740c-45b5-af6f-0518ef8493d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.574204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dc98e8-740c-45b5-af6f-0518ef8493d1 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.584083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1412/2825] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:43.588602 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.598422 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a15895-0978-443c-ac80-2656658a3954 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.603141 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.603762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.604035 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.604302 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] PATH_INFO: `/v3/projects/fc95d0325357419d90ca5205ec0eaec9/users/81cf11285d4349baa874e30ba94bd1d2/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.618190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc95d0325357419d90ca5205ec0eaec9, user_id=81cf11285d4349baa874e30ba94bd1d2, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.621074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.634173 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1414/2826] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:43 2022] PUT /identity/v3/projects/fc95d0325357419d90ca5205ec0eaec9/users/81cf11285d4349baa874e30ba94bd1d2/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:44:43.700016 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4f0f8-33ba-4a82-80e3-103e38279151 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.700299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4f0f8-33ba-4a82-80e3-103e38279151 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.700573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4f0f8-33ba-4a82-80e3-103e38279151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.728891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-16b4f0f8-33ba-4a82-80e3-103e38279151 None None] MFA Rules not processed for user `81cf11285d4349baa874e30ba94bd1d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:43.730128 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-16b4f0f8-33ba-4a82-80e3-103e38279151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:43.794402 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1413/2827] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:43.798808 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.839681 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:INSERT=5,keystone:DELETE=1,keystone:SELECT=94 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:43.866755 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e81c2c-719d-4a45-82c7-1156dab8da7b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.871838 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.872427 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.872700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.872966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.873585 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.875532 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:43.904005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1415/2828] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:43 2022] POST /identity/v3/projects => generated 375 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:43.908274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4f0f8-33ba-4a82-80e3-103e38279151 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:43.919476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4f0f8-33ba-4a82-80e3-103e38279151 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:43.968309 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:43.975194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:43.975981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:43.976306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:43.976715 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:43.977329 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:43.979258 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:44.000064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1414/2829] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:43 2022] POST /identity/v3/users => generated 294 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:44:44.004065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:44.015314 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6103b74c-b5b0-4c7f-912d-a46599e360b3 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:44.020323 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:44.020924 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:44.021230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:44.021503 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:44.022248 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:44.023970 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:44.035474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1416/2830] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:44:44.039801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:44.049604 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f1e92e-73bc-4aac-a766-26c187d942e2 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:44.054852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:44.055525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:44.055810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:44.056085 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] PATH_INFO: `/v3/projects/9222af7a60ab4dce9ad255090767c6e5/users/84ec783e7f504a9c9afc48d1a08d4aeb/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:44.071810 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9222af7a60ab4dce9ad255090767c6e5, user_id=84ec783e7f504a9c9afc48d1a08d4aeb, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:44.074436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:44.087198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1415/2831] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:44 2022] PUT /identity/v3/projects/9222af7a60ab4dce9ad255090767c6e5/users/84ec783e7f504a9c9afc48d1a08d4aeb/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:44.091016 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:44.101277 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fda992e-1763-4251-92c2-48b506cea60b admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:44.156551 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93be621a-47af-411d-8610-22b5d65209a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:44.157072 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be621a-47af-411d-8610-22b5d65209a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:44.157348 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be621a-47af-411d-8610-22b5d65209a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:44.157621 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93be621a-47af-411d-8610-22b5d65209a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:44.158251 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be621a-47af-411d-8610-22b5d65209a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:44.159857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93be621a-47af-411d-8610-22b5d65209a0 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:44.169792 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1417/2832] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:44 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:44.174075 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:44.184267 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbc9514-4b48-4abb-9905-665a87419f3d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:44.188819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:44.189456 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:44.189730 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:44.189998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] PATH_INFO: `/v3/projects/9222af7a60ab4dce9ad255090767c6e5/users/84ec783e7f504a9c9afc48d1a08d4aeb/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:44.191911 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9222af7a60ab4dce9ad255090767c6e5, user_id=84ec783e7f504a9c9afc48d1a08d4aeb, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:44.195094 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:44.204967 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1416/2833] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:44 2022] PUT /identity/v3/projects/9222af7a60ab4dce9ad255090767c6e5/users/84ec783e7f504a9c9afc48d1a08d4aeb/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:44.215693 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:44.215982 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:44.216259 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:44.244009 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] MFA Rules not processed for user `84ec783e7f504a9c9afc48d1a08d4aeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:44.245483 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:44.315412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1418/2834] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:44 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:44.327885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:44.329610 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:44.387174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6574121e-39a0-4a12-b130-22133000aa95 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:44.390727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:44.391414 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:44.391693 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:44.391966 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:44.447274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:44.448625 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:44.457230 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1417/2835] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:47.284412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:47.285800 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:47.320384 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:47.342922 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b862739d-2133-4b91-bab3-90acb7d5f0d6 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:47.346462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:47.347085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:47.347359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:47.347624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:47.349294 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:47.410548 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:47.411870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:47.421330 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1419/2836] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:47 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:48.097347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:48.106650 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c4ab30-6ba0-4fd5-8e1e-6a2209505a81 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:48.110244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:48.110845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:48.111149 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:48.111416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:48.169356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:48.170823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:48.179628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1418/2837] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:48 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:48.714308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:48.723974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748e4bb0-d334-4b19-8308-784e5af9d2c7 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:48.728260 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259da1b6-9bb1-43e3-a3ed-3971c531b75d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:48.728993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259da1b6-9bb1-43e3-a3ed-3971c531b75d service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:48.729272 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259da1b6-9bb1-43e3-a3ed-3971c531b75d service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:48.729524 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259da1b6-9bb1-43e3-a3ed-3971c531b75d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:48.738852 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259da1b6-9bb1-43e3-a3ed-3971c531b75d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:48.740286 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259da1b6-9bb1-43e3-a3ed-3971c531b75d service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:48.750694 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1420/2838] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:48 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:55.820453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:55.824609 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:55.884377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4797f76e-49c3-489d-bfac-1ee009c937a2 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:55.888845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:55.889457 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:55.889729 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:55.889997 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] PATH_INFO: `/v3/users/fcdf07e337cc470a89ad83025fba1a3b` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:55.891144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fcdf07e337cc470a89ad83025fba1a3b)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:55.893034 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:55.949517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fcdf07e337cc470a89ad83025fba1a3b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:55.950401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fcdf07e337cc470a89ad83025fba1a3b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:55.958495 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fcdf07e337cc470a89ad83025fba1a3b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:55.959246 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fcdf07e337cc470a89ad83025fba1a3b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:55.972555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fcdf07e337cc470a89ad83025fba1a3b'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:55.986193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1419/2839] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:55 2022] DELETE /identity/v3/users/fcdf07e337cc470a89ad83025fba1a3b => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:55.990669 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:55.991234 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:55.992029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:56.012281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:56.013474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:56.078340 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1421/2840] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:56.089025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:56.090424 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:56.148820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c91fc3-fce5-4a6f-8df6-c575355899d4 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:56.152371 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:56.152968 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:56.153244 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:56.153518 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:56.162275 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:56.164261 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:56.173395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1420/2841] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:56.439299 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:56.440816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:56.454460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6164a095-948f-4434-a5da-720b6c56bc89 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:56.459129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:56.459759 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:56.460039 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:56.460359 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] PATH_INFO: `/v3/projects/b1e5d4726c054a16b1f7c8e897432d8c` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:56.461660 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1e5d4726c054a16b1f7c8e897432d8c)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:56.463684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:56.510283 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1e5d4726c054a16b1f7c8e897432d8c', 'request_id': 'req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.516623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1e5d4726c054a16b1f7c8e897432d8c', 'request_id': 'req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.522571 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1e5d4726c054a16b1f7c8e897432d8c', 'request_id': 'req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.525136 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:56.537333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] The token cache is being invalidate because project b1e5d4726c054a16b1f7c8e897432d8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:56.537617 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.538388 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.539787 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1422/2842] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:56 2022] DELETE /identity/v3/projects/b1e5d4726c054a16b1f7c8e897432d8c => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:56.543779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:56.553193 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681fe24c-f563-47b1-8cd2-54e0b17b9676 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:56.574360 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:56.611906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:56.612532 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:56.612811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:56.613083 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] PATH_INFO: `/v3/users/81cf11285d4349baa874e30ba94bd1d2` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:56.614195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] RBAC: Authorizing `identity:delete_user(user_id=81cf11285d4349baa874e30ba94bd1d2)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:56.616159 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:56.665123 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81cf11285d4349baa874e30ba94bd1d2'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.665760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81cf11285d4349baa874e30ba94bd1d2'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.672590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81cf11285d4349baa874e30ba94bd1d2'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.673217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81cf11285d4349baa874e30ba94bd1d2'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.686256 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81cf11285d4349baa874e30ba94bd1d2'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:56.699994 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1421/2843] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:56 2022] DELETE /identity/v3/users/81cf11285d4349baa874e30ba94bd1d2 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:56.973276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:56.974741 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:56.989031 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6fbc1e-2c06-4922-9f04-3d91a75f6a80 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:57.057178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:57.057809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:57.058065 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:57.058334 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] PATH_INFO: `/v3/projects/fc95d0325357419d90ca5205ec0eaec9` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:57.059460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc95d0325357419d90ca5205ec0eaec9)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:57.061333 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:57.109130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc95d0325357419d90ca5205ec0eaec9', 'request_id': 'req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.115243 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc95d0325357419d90ca5205ec0eaec9', 'request_id': 'req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.122230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc95d0325357419d90ca5205ec0eaec9', 'request_id': 'req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.135539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] The token cache is being invalidate because project fc95d0325357419d90ca5205ec0eaec9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:57.135826 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.136459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.139022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1423/2844] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:56 2022] DELETE /identity/v3/projects/fc95d0325357419d90ca5205ec0eaec9 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:57.143474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:57.144934 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:57.160337 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6968482-ab50-40e2-8c55-458ad4550422 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:57.229356 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:57.229991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:57.230265 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:57.230535 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] PATH_INFO: `/v3/users/84ec783e7f504a9c9afc48d1a08d4aeb` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:57.231667 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=84ec783e7f504a9c9afc48d1a08d4aeb)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:57.233590 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:57.278078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:57.279790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84ec783e7f504a9c9afc48d1a08d4aeb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.280755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84ec783e7f504a9c9afc48d1a08d4aeb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.288980 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84ec783e7f504a9c9afc48d1a08d4aeb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.289624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84ec783e7f504a9c9afc48d1a08d4aeb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.301756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-7e8dc0f6-86b0-4882-b7d3-2d3b3c6eccb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84ec783e7f504a9c9afc48d1a08d4aeb'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.314466 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1422/2845] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:57 2022] DELETE /identity/v3/users/84ec783e7f504a9c9afc48d1a08d4aeb => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:57.848514 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:57.849946 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:57.862835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf84f7-9c8a-4491-89a6-5e8ba79b1c4c admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:57.921549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:57.922169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:57.922448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:57.922721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] PATH_INFO: `/v3/projects/9222af7a60ab4dce9ad255090767c6e5` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:57.923865 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9222af7a60ab4dce9ad255090767c6e5)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:57.925746 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:57.957158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:57.974835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9222af7a60ab4dce9ad255090767c6e5', 'request_id': 'req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.981677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9222af7a60ab4dce9ad255090767c6e5', 'request_id': 'req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.987146 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9222af7a60ab4dce9ad255090767c6e5', 'request_id': 'req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:57.999844 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] The token cache is being invalidate because project 9222af7a60ab4dce9ad255090767c6e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:44:58.000133 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:58.000736 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-ee62b6c7-2c4f-4cd2-b7a1-e445261158b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:58.002225 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1424/2846] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:44:57 2022] DELETE /identity/v3/projects/9222af7a60ab4dce9ad255090767c6e5 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:58.007055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aae4d9-c4ff-45d0-9c2e-0cc86f831be4 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.007347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aae4d9-c4ff-45d0-9c2e-0cc86f831be4 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.007623 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aae4d9-c4ff-45d0-9c2e-0cc86f831be4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.028736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f5aae4d9-c4ff-45d0-9c2e-0cc86f831be4 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:58.029844 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f5aae4d9-c4ff-45d0-9c2e-0cc86f831be4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:58.087747 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1423/2847] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:58.096667 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca5f6d4-ddcb-48d1-9e74-7953e9416cac None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.097163 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca5f6d4-ddcb-48d1-9e74-7953e9416cac None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.097274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca5f6d4-ddcb-48d1-9e74-7953e9416cac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.117134 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-cca5f6d4-ddcb-48d1-9e74-7953e9416cac None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:58.118319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-cca5f6d4-ddcb-48d1-9e74-7953e9416cac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:58.126527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1425/2848] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:58.130683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aae4d9-c4ff-45d0-9c2e-0cc86f831be4 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:58.140307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aae4d9-c4ff-45d0-9c2e-0cc86f831be4 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:58.144802 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:58.145401 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.145646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.145937 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.146572 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:44:58.146874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:58.149038 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:58.160312 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1424/2849] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:44:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 06 04:44:58.165069 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca5f6d4-ddcb-48d1-9e74-7953e9416cac None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:58.174140 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca5f6d4-ddcb-48d1-9e74-7953e9416cac None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:58.179000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:58.179738 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.180082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.180564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.181233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:58.183157 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:58.210610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1426/2850] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:44:58 2022] POST /identity/v3/projects => generated 391 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:58.214776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:58.227223 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78acb86-fb81-4b90-8a30-7da87cd2994a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:58.286379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:58.286895 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.287198 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.287474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.288097 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:58.290597 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:58.314950 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1425/2851] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:44:58 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:44:58.316358 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:58.336497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702e6489-6fd0-4280-869a-0956d0368446 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:58.341779 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:58.342425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.342718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.343051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.343686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:58.345295 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:58.356479 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1427/2852] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:58 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:44:58.360620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:58.371133 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdea799-3e41-4df5-b770-46c427c96017 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:58.375898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1150bb11-704a-4aff-9140-9ffb24c1248a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:58.376486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1150bb11-704a-4aff-9140-9ffb24c1248a admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.376760 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1150bb11-704a-4aff-9140-9ffb24c1248a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.377025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1150bb11-704a-4aff-9140-9ffb24c1248a admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.377655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1150bb11-704a-4aff-9140-9ffb24c1248a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:58.379408 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1150bb11-704a-4aff-9140-9ffb24c1248a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:58.391182 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1426/2853] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:44:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 06 04:44:58.395209 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:58.405825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb75221-0f88-4c61-94e1-8fb3525d0054 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:58.410572 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:58.411310 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.411654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.411979 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] PATH_INFO: `/v3/projects/53c1325f56874c97bf1efc218fcdaf4d/users/3a883bfb65aa42cdb8b78a0743a1d006/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.435094 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] RBAC: Authorizing `identity:create_grant(project_id=53c1325f56874c97bf1efc218fcdaf4d, user_id=3a883bfb65aa42cdb8b78a0743a1d006, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:58.437858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:58.451190 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1428/2854] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:44:58 2022] PUT /identity/v3/projects/53c1325f56874c97bf1efc218fcdaf4d/users/3a883bfb65aa42cdb8b78a0743a1d006/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:58.462144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.462421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.462733 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.489670 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] MFA Rules not processed for user `3a883bfb65aa42cdb8b78a0743a1d006`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:58.490856 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:58.507453 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:58.555993 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1427/2855] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:58 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:44:58.567223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:58.569045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:58.626984 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e61d02-7b66-4788-ba10-ae782f3871df admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:58.630846 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f57f798-8290-4b1e-a57b-2acaf1d6e840 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:58.631541 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f57f798-8290-4b1e-a57b-2acaf1d6e840 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:58.631819 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f57f798-8290-4b1e-a57b-2acaf1d6e840 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:58.632588 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f57f798-8290-4b1e-a57b-2acaf1d6e840 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:58.641056 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f57f798-8290-4b1e-a57b-2acaf1d6e840 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:58.642438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f57f798-8290-4b1e-a57b-2acaf1d6e840 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:58.652818 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1429/2856] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:58 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:44:59.565972 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:59.567416 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:59.633316 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1d2437-9f44-4229-b166-e3f390819cd4 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:59.637986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:59.638601 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:59.638894 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:59.639175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] PATH_INFO: `/v3/users/fba90caacf3e4a739f854e3ac0b090fd` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:59.640266 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] RBAC: Authorizing `identity:delete_user(user_id=fba90caacf3e4a739f854e3ac0b090fd)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:59.642439 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:59.687298 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fba90caacf3e4a739f854e3ac0b090fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:59.687935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fba90caacf3e4a739f854e3ac0b090fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:59.694668 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fba90caacf3e4a739f854e3ac0b090fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:59.695286 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fba90caacf3e4a739f854e3ac0b090fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:59.706936 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fba90caacf3e4a739f854e3ac0b090fd'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:44:59.720174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1428/2857] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:44:59 2022] DELETE /identity/v3/users/fba90caacf3e4a739f854e3ac0b090fd => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:44:59.724624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:59.725092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:59.725207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:59.744452 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:44:59.745896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:59.792169 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:59.805077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1430/2858] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:44:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 06 04:44:59.814573 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:44:59.815988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:44:59.858029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:44:59.877755 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3549e8-2a74-43aa-9d62-4ed9bb65c59c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:44:59.882396 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4689a75c-1962-47d6-adad-924fd3b7552d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:44:59.883202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4689a75c-1962-47d6-adad-924fd3b7552d service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:44:59.883478 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4689a75c-1962-47d6-adad-924fd3b7552d service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:44:59.883745 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4689a75c-1962-47d6-adad-924fd3b7552d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:44:59.892542 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4689a75c-1962-47d6-adad-924fd3b7552d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:44:59.893952 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4689a75c-1962-47d6-adad-924fd3b7552d service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:44:59.903248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1429/2859] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:44:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:45:00.180734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.185568 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:00.197586 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db5c25f-d8bf-4842-840a-87c114026c70 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.202354 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.203317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.203317 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.203565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] PATH_INFO: `/v3/projects/e93f474e93ce417f9a81c5c496687a71` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.204628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e93f474e93ce417f9a81c5c496687a71)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.206516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.265018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e93f474e93ce417f9a81c5c496687a71', 'request_id': 'req-414aebcc-6b18-4b32-9805-1eaacd055cfb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:00.271088 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e93f474e93ce417f9a81c5c496687a71', 'request_id': 'req-414aebcc-6b18-4b32-9805-1eaacd055cfb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:00.276795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e93f474e93ce417f9a81c5c496687a71', 'request_id': 'req-414aebcc-6b18-4b32-9805-1eaacd055cfb'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:00.288942 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] The token cache is being invalidate because project e93f474e93ce417f9a81c5c496687a71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:45:00.289223 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:00.289836 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-414aebcc-6b18-4b32-9805-1eaacd055cfb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:00.291335 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1431/2860] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:45:00 2022] DELETE /identity/v3/projects/e93f474e93ce417f9a81c5c496687a71 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:00.296188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2305b2-1867-4a41-916b-86043de51f9f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.296486 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2305b2-1867-4a41-916b-86043de51f9f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.296765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2305b2-1867-4a41-916b-86043de51f9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.316436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9e2305b2-1867-4a41-916b-86043de51f9f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:00.317635 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9e2305b2-1867-4a41-916b-86043de51f9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:00.376152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1430/2861] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:00.385279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b542b388-0e1a-472e-aa1f-7ae7d15c8bab None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.385565 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b542b388-0e1a-472e-aa1f-7ae7d15c8bab None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.385841 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b542b388-0e1a-472e-aa1f-7ae7d15c8bab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.405823 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-b542b388-0e1a-472e-aa1f-7ae7d15c8bab None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:00.407037 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b542b388-0e1a-472e-aa1f-7ae7d15c8bab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:00.415624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1432/2862] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:00.419870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2305b2-1867-4a41-916b-86043de51f9f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.429525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2305b2-1867-4a41-916b-86043de51f9f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.434078 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.434669 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.434948 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.435259 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.435900 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:45:00.436205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.437791 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.449589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1431/2863] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:45:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:45:00.453631 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b542b388-0e1a-472e-aa1f-7ae7d15c8bab None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.463491 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b542b388-0e1a-472e-aa1f-7ae7d15c8bab None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.468105 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.468692 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.468962 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.469230 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.469857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.471770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.499070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1433/2864] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:45:00 2022] POST /identity/v3/projects => generated 363 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:45:00.503025 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.513065 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf4e37a-4d75-4f77-9e43-d33284a9e15b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.568817 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.569998 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.570282 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.570628 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.571299 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.573206 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.594135 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1432/2865] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:45:00 2022] POST /identity/v3/users => generated 282 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:45:00.598164 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.608376 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a38e201-de96-493b-bf4d-025b26ba2b91 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.612885 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.613457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.613730 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.613996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.614624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.616261 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.626184 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1434/2866] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:00 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:00.629953 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.640278 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1f84fb-eecc-4cf2-87e9-c7d0c5fb230a admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.644909 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ab629a-54a1-4f0f-9aa5-7f1e49561fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.645489 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ab629a-54a1-4f0f-9aa5-7f1e49561fdf admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.645768 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ab629a-54a1-4f0f-9aa5-7f1e49561fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.646036 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ab629a-54a1-4f0f-9aa5-7f1e49561fdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.646665 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ab629a-54a1-4f0f-9aa5-7f1e49561fdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.649646 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ab629a-54a1-4f0f-9aa5-7f1e49561fdf admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.661006 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1433/2867] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:00.665830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.675549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0e0836-179b-4722-9a2c-7780fb1090c6 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.680780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.681410 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.681682 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.681952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] PATH_INFO: `/v3/projects/65726029975f447fb5ba678331592156/users/07d81d8f619644ce9e94b243ae323b11/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.696130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=65726029975f447fb5ba678331592156, user_id=07d81d8f619644ce9e94b243ae323b11, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.698721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.711654 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1435/2868] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:45:00 2022] PUT /identity/v3/projects/65726029975f447fb5ba678331592156/users/07d81d8f619644ce9e94b243ae323b11/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:00.721962 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.722243 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.722511 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.752395 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] MFA Rules not processed for user `07d81d8f619644ce9e94b243ae323b11`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:00.753540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:00.786010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:00.820220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1434/2869] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:00 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:00.831594 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:00.833276 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:00.889247 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee68bf-6dc1-4d43-b0b6-1ac82c946f5d admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:00.892857 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c51ab67-20f5-43a4-a2bf-634d775fa161 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:00.893444 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c51ab67-20f5-43a4-a2bf-634d775fa161 service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:00.893721 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c51ab67-20f5-43a4-a2bf-634d775fa161 service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:00.893991 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c51ab67-20f5-43a4-a2bf-634d775fa161 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:00.903347 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c51ab67-20f5-43a4-a2bf-634d775fa161 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:00.904758 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c51ab67-20f5-43a4-a2bf-634d775fa161 service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:00.913675 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1436/2870] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:45:00 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:45:10.816591 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=11 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:10.909441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=83,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:24.151886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:24.153250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:24.223528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c659ea02-40e4-46d6-b480-c13fde40a1d4 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:24.228350 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:24.228887 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:24.229167 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:24.229438 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] PATH_INFO: `/v3/users/3a883bfb65aa42cdb8b78a0743a1d006` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:24.230567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a883bfb65aa42cdb8b78a0743a1d006)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:24.232519 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:24.278525 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a883bfb65aa42cdb8b78a0743a1d006'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.279195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a883bfb65aa42cdb8b78a0743a1d006'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.285846 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a883bfb65aa42cdb8b78a0743a1d006'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.286458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a883bfb65aa42cdb8b78a0743a1d006'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.298015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a883bfb65aa42cdb8b78a0743a1d006'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.310276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1435/2871] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:45:24 2022] DELETE /identity/v3/users/3a883bfb65aa42cdb8b78a0743a1d006 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:24.315034 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:24.315549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:24.315549 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:24.335462 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:24.336651 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:24.396756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1437/2872] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:24.407362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:24.408702 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:24.464898 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1aa2d2-8ff2-4c62-a4ce-9ef18d04f30b admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:24.468567 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fee76cb-1075-410e-9c9a-b54df668b107 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:24.469161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee76cb-1075-410e-9c9a-b54df668b107 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:24.469436 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee76cb-1075-410e-9c9a-b54df668b107 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:24.469703 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fee76cb-1075-410e-9c9a-b54df668b107 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:24.478885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fee76cb-1075-410e-9c9a-b54df668b107 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:24.480351 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fee76cb-1075-410e-9c9a-b54df668b107 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:24.490490 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1436/2873] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:45:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:45:24.757589 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:24.760174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:24.773048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c9913-6dc8-4a9e-b98b-ef77f204a40e None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:24.778370 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:24.779025 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:24.779320 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:24.779577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] PATH_INFO: `/v3/projects/53c1325f56874c97bf1efc218fcdaf4d` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:24.780727 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] RBAC: Authorizing `identity:delete_project(project_id=53c1325f56874c97bf1efc218fcdaf4d)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:24.782627 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:24.832521 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53c1325f56874c97bf1efc218fcdaf4d', 'request_id': 'req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.839327 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53c1325f56874c97bf1efc218fcdaf4d', 'request_id': 'req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.844873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53c1325f56874c97bf1efc218fcdaf4d', 'request_id': 'req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.857403 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] The token cache is being invalidate because project 53c1325f56874c97bf1efc218fcdaf4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:45:24.857686 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.858431 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-6a60d6fc-fb61-4bfa-bc75-2c090c4e205b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:24.859938 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1438/2874] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:45:24 2022] DELETE /identity/v3/projects/53c1325f56874c97bf1efc218fcdaf4d => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:24.864838 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93454549-2ae4-40c7-a3dd-1412ebfc32d3 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:24.865125 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93454549-2ae4-40c7-a3dd-1412ebfc32d3 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:24.865400 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93454549-2ae4-40c7-a3dd-1412ebfc32d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:24.891497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-93454549-2ae4-40c7-a3dd-1412ebfc32d3 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:24.892690 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-93454549-2ae4-40c7-a3dd-1412ebfc32d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:24.909054 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:24.953872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1437/2875] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:24.963211 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06277e27-d9b4-4067-ac0a-2111641c0f10 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:24.963497 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06277e27-d9b4-4067-ac0a-2111641c0f10 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:24.963770 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06277e27-d9b4-4067-ac0a-2111641c0f10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:24.984175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-06277e27-d9b4-4067-ac0a-2111641c0f10 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:24.985374 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-06277e27-d9b4-4067-ac0a-2111641c0f10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:24.993153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1439/2876] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:24.998174 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93454549-2ae4-40c7-a3dd-1412ebfc32d3 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:25.007868 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93454549-2ae4-40c7-a3dd-1412ebfc32d3 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:25.013248 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:25.013840 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.014118 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.014390 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] PATH_INFO: `/v3/domains` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.015067 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81400) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:45:25.015370 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:25.016988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:25.029175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1438/2877] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:45:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:45:25.033820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06277e27-d9b4-4067-ac0a-2111641c0f10 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:25.044122 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06277e27-d9b4-4067-ac0a-2111641c0f10 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:25.048853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:25.049441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.049712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.050116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.050743 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:25.052652 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:25.080734 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1440/2878] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:45:25 2022] POST /identity/v3/projects => generated 379 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:45:25.092641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:25.097858 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f03c82-6231-434f-8d9e-3e45eebbc075 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:25.158474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:25.158474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.158474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.158474 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.159347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:25.159347 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:25.179183 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1439/2879] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:45:25 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:45:25.183968 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:25.195296 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ced11-3dcc-47b3-983e-b6a4359f44f4 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:25.202421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:25.203232 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.203516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.203820 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.204512 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:25.206045 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:25.217778 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1441/2880] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:25.222166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:25.236874 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f19b79-81fb-480d-8746-402097519a34 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:25.241717 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceeb55e-8977-4ab3-a643-a05f7e28a0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:25.242388 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceeb55e-8977-4ab3-a643-a05f7e28a0b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.242732 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceeb55e-8977-4ab3-a643-a05f7e28a0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.243120 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceeb55e-8977-4ab3-a643-a05f7e28a0b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.243905 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceeb55e-8977-4ab3-a643-a05f7e28a0b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:25.245818 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceeb55e-8977-4ab3-a643-a05f7e28a0b1 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:25.257195 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1440/2881] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:25 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:25.262474 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:25.273012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e41ce6-454a-4c64-8bdf-8a0f21694aa9 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:25.278911 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:25.279582 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] REQUEST_METHOD: `PUT` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.279863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.280137 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] PATH_INFO: `/v3/projects/a707ca1042dd4fc99ff1dbd033efe0e8/users/450dcd00d2404852b9635e72650d0c7f/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.294936 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a707ca1042dd4fc99ff1dbd033efe0e8, user_id=450dcd00d2404852b9635e72650d0c7f, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:25.297567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:25.311263 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1442/2882] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:45:25 2022] PUT /identity/v3/projects/a707ca1042dd4fc99ff1dbd033efe0e8/users/450dcd00d2404852b9635e72650d0c7f/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:25.316245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dafa866-9cf0-4ad2-9f31-28568bdbe0bd None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.316531 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dafa866-9cf0-4ad2-9f31-28568bdbe0bd None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.316831 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dafa866-9cf0-4ad2-9f31-28568bdbe0bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.336072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1dafa866-9cf0-4ad2-9f31-28568bdbe0bd None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:25.337339 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1dafa866-9cf0-4ad2-9f31-28568bdbe0bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:25.396119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1441/2883] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:25.405921 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:25.407637 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:25.429597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:25.468853 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8793f0-e50b-4f3c-8686-ee3704086563 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:25.472468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:25.473076 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:25.473369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:25.473636 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:25.482871 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:25.484361 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:25.494351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1443/2884] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:45:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:45:29.469343 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9344fee8-1012-4e94-9193-4d64aa31fa38 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:29.469873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9344fee8-1012-4e94-9193-4d64aa31fa38 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:29.469873 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9344fee8-1012-4e94-9193-4d64aa31fa38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:29.508056 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-9344fee8-1012-4e94-9193-4d64aa31fa38 None None] MFA Rules not processed for user `450dcd00d2404852b9635e72650d0c7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:29.509783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-9344fee8-1012-4e94-9193-4d64aa31fa38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:29.530064 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:29.583523 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1442/2885] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:29 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:29.589274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:29.599744 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb60301b-1d4e-49fb-9687-4ba9cbcd5569 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:29.604980 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:29.605690 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:29.605993 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:29.606266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:29.607007 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:29.609319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:29.646612 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1444/2886] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:45:29 2022] POST /identity/v3/projects => generated 377 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:45:29.647744 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9344fee8-1012-4e94-9193-4d64aa31fa38 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:29.660200 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9344fee8-1012-4e94-9193-4d64aa31fa38 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:29.731872 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:29.732634 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:29.732916 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:29.733188 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] PATH_INFO: `/v3/users` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:29.733832 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:29.735878 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:29.762499 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1443/2887] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:45:29 2022] POST /identity/v3/users => generated 295 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:45:29.766725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:29.779854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb6139-aa21-44e6-b29a-70a468045bd2 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:29.784577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:29.785266 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:29.785544 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:29.785816 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:29.786467 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:29.788129 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:29.799459 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1445/2888] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:29.803736 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:29.816727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3807f-c4df-46a9-b465-b0fe20a37d08 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:29.821692 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:29.822441 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:29.822720 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:29.823015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] PATH_INFO: `/v3/projects/31adbae4cbd54ad0992fdaeb77ea0e56/users/0fcbe8f22f6340f68db571f9ea06c21f/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:29.840046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] RBAC: Authorizing `identity:create_grant(project_id=31adbae4cbd54ad0992fdaeb77ea0e56, user_id=0fcbe8f22f6340f68db571f9ea06c21f, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:29.843175 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:29.860022 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1444/2889] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:45:29 2022] PUT /identity/v3/projects/31adbae4cbd54ad0992fdaeb77ea0e56/users/0fcbe8f22f6340f68db571f9ea06c21f/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:29.863923 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:29.875650 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c92e24e-7569-4a9b-9e7f-5553a7a5da0e admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:29.939285 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad59080-cbcf-481d-8a4f-c373e4a76698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:29.939897 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad59080-cbcf-481d-8a4f-c373e4a76698 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:29.940175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad59080-cbcf-481d-8a4f-c373e4a76698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:29.940446 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad59080-cbcf-481d-8a4f-c373e4a76698 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:29.941085 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad59080-cbcf-481d-8a4f-c373e4a76698 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:29.942661 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad59080-cbcf-481d-8a4f-c373e4a76698 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:29.952981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1446/2890] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:29 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:29.957641 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:29.970751 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6838ad4d-f29f-4ceb-a78e-24d1fc3f983f admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:29.975823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:29.976642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:29.976977 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:29.977196 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] PATH_INFO: `/v3/projects/31adbae4cbd54ad0992fdaeb77ea0e56/users/0fcbe8f22f6340f68db571f9ea06c21f/roles/496e1df1a0c74115a98f6bbb36196a95` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:29.979811 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] RBAC: Authorizing `identity:create_grant(project_id=31adbae4cbd54ad0992fdaeb77ea0e56, user_id=0fcbe8f22f6340f68db571f9ea06c21f, role_id=496e1df1a0c74115a98f6bbb36196a95)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:29.982507 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:29.994046 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1445/2891] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:45:29 2022] PUT /identity/v3/projects/31adbae4cbd54ad0992fdaeb77ea0e56/users/0fcbe8f22f6340f68db571f9ea06c21f/roles/496e1df1a0c74115a98f6bbb36196a95 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:35.198653 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:35.199790 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:35.200357 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:35.248771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] MFA Rules not processed for user `0fcbe8f22f6340f68db571f9ea06c21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:35.250077 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:35.360511 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1447/2892] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:35 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:35.385578 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:35.387412 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:35.507624 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4c3391-a8f4-4646-848a-345d2e75751e admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:35.513986 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4d47c5-c4cd-4918-8e5e-54184bbfaceb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:35.514757 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4d47c5-c4cd-4918-8e5e-54184bbfaceb service nova] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:35.515108 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4d47c5-c4cd-4918-8e5e-54184bbfaceb service nova] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:35.515434 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4d47c5-c4cd-4918-8e5e-54184bbfaceb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:35.522949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:35.604728 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4d47c5-c4cd-4918-8e5e-54184bbfaceb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:35.606352 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4d47c5-c4cd-4918-8e5e-54184bbfaceb service nova] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:35.617867 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1446/2893] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:45:35 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:45:45.357096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=94,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:45.613225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=24 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:56.471996 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:56.473425 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:56.537706 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5344afcb-6cd1-4096-9bea-8c9bf5b42cf2 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:56.542726 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:56.543407 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:56.543688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:56.543956 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] PATH_INFO: `/v3/users/07d81d8f619644ce9e94b243ae323b11` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:56.545071 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=07d81d8f619644ce9e94b243ae323b11)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:56.546981 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:56.593226 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07d81d8f619644ce9e94b243ae323b11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:56.593873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07d81d8f619644ce9e94b243ae323b11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:56.600658 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07d81d8f619644ce9e94b243ae323b11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:56.612676 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07d81d8f619644ce9e94b243ae323b11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:56.613297 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07d81d8f619644ce9e94b243ae323b11'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:56.626484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1448/2894] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:45:56 2022] DELETE /identity/v3/users/07d81d8f619644ce9e94b243ae323b11 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:56.631058 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:56.631553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:56.631655 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:56.650886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:56.652647 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:56.715852 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1447/2895] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:56.725460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:56.726851 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:56.784308 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a17d19-ced9-47a1-8cac-fcbb9e94c3ac admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:56.788116 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2a45b8-c855-4937-97a9-00be3af82db8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:56.788700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2a45b8-c855-4937-97a9-00be3af82db8 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:56.788974 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2a45b8-c855-4937-97a9-00be3af82db8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:56.789239 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2a45b8-c855-4937-97a9-00be3af82db8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:56.798863 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2a45b8-c855-4937-97a9-00be3af82db8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:56.800319 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2a45b8-c855-4937-97a9-00be3af82db8 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:56.809677 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1449/2896] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:45:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:45:57.079105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.080617 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:57.095105 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6a90d3-7589-4276-9ed2-ccb5d737da6f None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.100283 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.101096 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.101372 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.101657 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] PATH_INFO: `/v3/projects/65726029975f447fb5ba678331592156` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.102863 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65726029975f447fb5ba678331592156)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.104889 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.151383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65726029975f447fb5ba678331592156', 'request_id': 'req-4a8cf208-5054-4bff-8880-812e4bd36cf8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:57.157580 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65726029975f447fb5ba678331592156', 'request_id': 'req-4a8cf208-5054-4bff-8880-812e4bd36cf8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:57.163750 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65726029975f447fb5ba678331592156', 'request_id': 'req-4a8cf208-5054-4bff-8880-812e4bd36cf8'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:57.176528 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] The token cache is being invalidate because project 65726029975f447fb5ba678331592156 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:45:57.176806 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:57.177421 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4a8cf208-5054-4bff-8880-812e4bd36cf8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:45:57.178884 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1448/2897] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:45:57 2022] DELETE /identity/v3/projects/65726029975f447fb5ba678331592156 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:57.183955 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58e622-22f4-4c2e-8b3c-9a5b5ed21701 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.184300 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58e622-22f4-4c2e-8b3c-9a5b5ed21701 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.184644 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a58e622-22f4-4c2e-8b3c-9a5b5ed21701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.204558 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-4a58e622-22f4-4c2e-8b3c-9a5b5ed21701 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:57.205733 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-4a58e622-22f4-4c2e-8b3c-9a5b5ed21701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:57.223005 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:57.268257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1450/2898] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:57.276415 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c76864f-e02f-43d9-af5f-f92dd6e283de None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.277363 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c76864f-e02f-43d9-af5f-f92dd6e283de None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.278042 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c76864f-e02f-43d9-af5f-f92dd6e283de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.302533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-7c76864f-e02f-43d9-af5f-f92dd6e283de None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:57.303788 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-7c76864f-e02f-43d9-af5f-f92dd6e283de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:57.312166 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1449/2899] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:57.316464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a58e622-22f4-4c2e-8b3c-9a5b5ed21701 None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.326476 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a58e622-22f4-4c2e-8b3c-9a5b5ed21701 None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.330985 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.331567 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.331839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.332103 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.332729 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81399) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 06 04:45:57.333022 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.334580 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.346440 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1451/2900] 10.222.0.33 () {58 vars in 1153 bytes} [Thu Oct 6 04:45:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 06 04:45:57.350748 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c76864f-e02f-43d9-af5f-f92dd6e283de None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.360458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c76864f-e02f-43d9-af5f-f92dd6e283de None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.365095 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.365683 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.365961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.366233 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] PATH_INFO: `/v3/projects` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.366866 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.368807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.397464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1450/2901] 10.222.0.33 () {60 vars in 1153 bytes} [Thu Oct 6 04:45:57 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 06 04:45:57.401652 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.412217 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef5280d-5cd4-4934-a24f-53cfa0901c89 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.467509 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.468093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.468369 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.468639 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] PATH_INFO: `/v3/users` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.469278 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.471178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.492093 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1452/2902] 10.222.0.33 () {60 vars in 1144 bytes} [Thu Oct 6 04:45:57 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 06 04:45:57.497074 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.507205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235bff17-b88b-4448-987d-798acce3136d admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.513104 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.514320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.514939 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.515517 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.517180 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.518784 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.529991 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1451/2903] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:57.534053 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.544281 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fec4f9-f642-4ecc-b978-327657c291ef admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.548835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc8117d-0021-4fd9-8e1d-d9ee579238d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.549412 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc8117d-0021-4fd9-8e1d-d9ee579238d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.549685 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc8117d-0021-4fd9-8e1d-d9ee579238d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.549966 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc8117d-0021-4fd9-8e1d-d9ee579238d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.550597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc8117d-0021-4fd9-8e1d-d9ee579238d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.552200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc8117d-0021-4fd9-8e1d-d9ee579238d3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.563070 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1453/2904] 10.222.0.33 () {58 vars in 1122 bytes} [Thu Oct 6 04:45:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 06 04:45:57.567031 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.577288 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfd913d-9e5b-4f33-9c25-31c45f9e88e2 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.581758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.582387 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.582660 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.582928 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] PATH_INFO: `/v3/projects/204d70df1e5a4324b989e55ad29c404c/users/74a0b9f681104da7836d7d68f1afbe0b/roles/68ada91d9625461ea199ad5a616e8f01` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.597990 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=204d70df1e5a4324b989e55ad29c404c, user_id=74a0b9f681104da7836d7d68f1afbe0b, role_id=68ada91d9625461ea199ad5a616e8f01)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.600534 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.612492 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1452/2905] 10.222.0.33 () {60 vars in 1483 bytes} [Thu Oct 6 04:45:57 2022] PUT /identity/v3/projects/204d70df1e5a4324b989e55ad29c404c/users/74a0b9f681104da7836d7d68f1afbe0b/roles/68ada91d9625461ea199ad5a616e8f01 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:45:57.623390 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f735fdb-a6fb-4773-9fe3-719b913f4a0c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.623671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f735fdb-a6fb-4773-9fe3-719b913f4a0c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.623940 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f735fdb-a6fb-4773-9fe3-719b913f4a0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.652096 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-7f735fdb-a6fb-4773-9fe3-719b913f4a0c None None] MFA Rules not processed for user `74a0b9f681104da7836d7d68f1afbe0b`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:45:57.653341 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-7f735fdb-a6fb-4773-9fe3-719b913f4a0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:57.716106 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1454/2906] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:45:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:45:57.726626 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:45:57.728353 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:45:57.750497 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:45:57.787082 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fbed85-23b0-4ca3-9fc8-52228fc00dd6 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:45:57.790560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:45:57.791181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:45:57.791455 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:45:57.791727 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:45:57.802582 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:45:57.805320 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:45:57.815160 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1453/2907] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:45:57 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:46:00.527608 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390ddbdd-1fe5-4bc3-9332-fa9332aa834c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:46:00.528109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390ddbdd-1fe5-4bc3-9332-fa9332aa834c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:46:00.528227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390ddbdd-1fe5-4bc3-9332-fa9332aa834c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:46:00.546904 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-390ddbdd-1fe5-4bc3-9332-fa9332aa834c None None] MFA Rules not processed for user `450dcd00d2404852b9635e72650d0c7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:46:00.548115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-390ddbdd-1fe5-4bc3-9332-fa9332aa834c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:00.562803 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:46:00.609372 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1455/2908] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:46:00 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:46:01.882862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:46:01.884274 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:01.953611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ed4bb-f04d-49df-8b4e-875311994921 service cinder] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:46:01.958629 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9302a2d-247e-49bc-9148-1d3815ab092c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:46:01.959290 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9302a2d-247e-49bc-9148-1d3815ab092c service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:46:01.959560 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9302a2d-247e-49bc-9148-1d3815ab092c service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:46:01.959826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9302a2d-247e-49bc-9148-1d3815ab092c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:46:01.961611 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d9302a2d-247e-49bc-9148-1d3815ab092c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:01.974107 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9302a2d-247e-49bc-9148-1d3815ab092c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:46:01.975718 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9302a2d-247e-49bc-9148-1d3815ab092c service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:46:01.986500 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1454/2909] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:46:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:46:08.009825 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390ddbdd-1fe5-4bc3-9332-fa9332aa834c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:46:08.071477 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390ddbdd-1fe5-4bc3-9332-fa9332aa834c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:46:08.076657 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:46:08.077275 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:46:08.077578 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:46:08.077849 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] PATH_INFO: `/v3/users/74a0b9f681104da7836d7d68f1afbe0b` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:46:08.079073 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74a0b9f681104da7836d7d68f1afbe0b)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:46:08.081047 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:46:08.126051 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74a0b9f681104da7836d7d68f1afbe0b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.126688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74a0b9f681104da7836d7d68f1afbe0b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.133880 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74a0b9f681104da7836d7d68f1afbe0b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.146012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74a0b9f681104da7836d7d68f1afbe0b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.146633 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74a0b9f681104da7836d7d68f1afbe0b'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.159090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1456/2910] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:46:08 2022] DELETE /identity/v3/users/74a0b9f681104da7836d7d68f1afbe0b => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:46:08.163581 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:46:08.164091 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:46:08.164205 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:46:08.183235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] MFA Rules not processed for user `f6cec6cd063d4a0684ad9f283f12b406`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:46:08.184425 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:08.244276 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1455/2911] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:46:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:46:08.254448 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:46:08.255937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:08.302546 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:46:08.315688 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536ebc4-dfcb-49a5-b8e4-ccc85dd6de62 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:46:08.319305 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:46:08.319896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:46:08.320172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:46:08.320438 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:46:08.329928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:46:08.331303 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:46:08.341464 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1457/2912] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:46:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:46:08.611072 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:46:08.612814 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:08.627826 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b123ba62-46a8-4fbd-908c-946ea2e6ca39 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:46:08.633057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:46:08.633779 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:46:08.634117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:46:08.634452 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] PATH_INFO: `/v3/projects/204d70df1e5a4324b989e55ad29c404c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:46:08.635672 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=204d70df1e5a4324b989e55ad29c404c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:46:08.638357 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:46:08.686071 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:46:08.687642 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '204d70df1e5a4324b989e55ad29c404c', 'request_id': 'req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.694171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '204d70df1e5a4324b989e55ad29c404c', 'request_id': 'req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.700739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '204d70df1e5a4324b989e55ad29c404c', 'request_id': 'req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.712815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] The token cache is being invalidate because project 204d70df1e5a4324b989e55ad29c404c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:46:08.713119 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.714250 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:46:08.715777 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1456/2913] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:46:08 2022] DELETE /identity/v3/projects/204d70df1e5a4324b989e55ad29c404c => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:46:18.194623 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:46:18.196131 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:18.207600 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd26e49e-3a75-454f-8b3f-6f6230111531 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:46:18.256329 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:46:18.256928 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:46:18.257200 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:46:18.257465 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:46:18.259177 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:46:18.318756 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:46:18.320195 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:46:18.329618 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1458/2914] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:46:18 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:46:18.709663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=7,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:46:28.325420 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=57 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:47:29.255654 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:47:29.257068 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:47:29.269277 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7cfb2-b34d-4b47-9b44-934eddf07ee0 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:47:29.272819 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff62e81-8571-48df-920a-93d0fc776e47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:47:29.273405 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff62e81-8571-48df-920a-93d0fc776e47 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:29.273681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff62e81-8571-48df-920a-93d0fc776e47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:29.273949 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff62e81-8571-48df-920a-93d0fc776e47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:29.276620 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-6ff62e81-8571-48df-920a-93d0fc776e47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:47:29.346013 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff62e81-8571-48df-920a-93d0fc776e47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:47:29.347392 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff62e81-8571-48df-920a-93d0fc776e47 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:47:29.358247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1457/2915] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:47:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:47:30.543227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:47:30.544695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:47:30.555380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b02494-4e06-40b5-9b13-bb75755d629e service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:47:30.559577 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c417dd-eb28-4c26-b63d-3c39153f3457 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:47:30.560165 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c417dd-eb28-4c26-b63d-3c39153f3457 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:30.560441 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c417dd-eb28-4c26-b63d-3c39153f3457 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:30.560712 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c417dd-eb28-4c26-b63d-3c39153f3457 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:30.562380 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-c6c417dd-eb28-4c26-b63d-3c39153f3457 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:47:30.575870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c417dd-eb28-4c26-b63d-3c39153f3457 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:47:30.577264 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c417dd-eb28-4c26-b63d-3c39153f3457 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:47:30.589098 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1459/2916] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:47:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:47:39.353485 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=34 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:47:40.583240 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=6 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:47:43.704362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f3cb1-d280-4bd0-baf4-da5280cc8ed6 None None] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:43.705101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f3cb1-d280-4bd0-baf4-da5280cc8ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:43.705101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f3cb1-d280-4bd0-baf4-da5280cc8ed6 None None] PATH_INFO: `/` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:43.706281 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1458/2917] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:47:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:47:43.713821 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d82ab-9b26-4227-a99b-e950df269283 None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:43.714460 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d82ab-9b26-4227-a99b-e950df269283 None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:43.714564 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d82ab-9b26-4227-a99b-e950df269283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:43.761626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-6e8d82ab-9b26-4227-a99b-e950df269283 None None] MFA Rules not processed for user `87829678318948f3b81bd107076bdfa7`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:47:43.763171 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-6e8d82ab-9b26-4227-a99b-e950df269283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:47:43.772725 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1460/2918] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:47:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:47:43.787235 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f3cb1-d280-4bd0-baf4-da5280cc8ed6 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:47:43.801442 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f3cb1-d280-4bd0-baf4-da5280cc8ed6 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:47:43.805362 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec03e868-4464-4368-b190-7aa9147061ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:47:43.806059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec03e868-4464-4368-b190-7aa9147061ce service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:43.806506 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec03e868-4464-4368-b190-7aa9147061ce service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:43.806845 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec03e868-4464-4368-b190-7aa9147061ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:43.814723 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec03e868-4464-4368-b190-7aa9147061ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:47:43.816315 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec03e868-4464-4368-b190-7aa9147061ce service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:47:43.825103 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1459/2919] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:47:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:47:43.907396 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09544022-7576-46e7-aa5b-22acf535cefb None None] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:43.907679 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09544022-7576-46e7-aa5b-22acf535cefb None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:43.907952 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09544022-7576-46e7-aa5b-22acf535cefb None None] PATH_INFO: `/` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:43.909279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1461/2920] 10.222.0.33 () {54 vars in 921 bytes} [Thu Oct 6 04:47:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 06 04:47:43.914181 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1056753-ae89-4bc4-a36c-18c53a5eb5a9 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:43.914464 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1056753-ae89-4bc4-a36c-18c53a5eb5a9 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:43.915487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1056753-ae89-4bc4-a36c-18c53a5eb5a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:43.951961 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-f1056753-ae89-4bc4-a36c-18c53a5eb5a9 None None] MFA Rules not processed for user `c8d2db883de44964b6e15b85a97726b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:47:43.953150 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-f1056753-ae89-4bc4-a36c-18c53a5eb5a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:47:43.962735 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1460/2921] 10.222.0.33 () {58 vars in 1018 bytes} [Thu Oct 6 04:47:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:47:43.973092 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09544022-7576-46e7-aa5b-22acf535cefb None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:47:43.984399 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09544022-7576-46e7-aa5b-22acf535cefb None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:47:43.988014 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2e8539-925f-4399-9379-be96886dedc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:47:43.988597 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2e8539-925f-4399-9379-be96886dedc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:47:43.988870 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2e8539-925f-4399-9379-be96886dedc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:47:43.989138 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2e8539-925f-4399-9379-be96886dedc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:47:43.999950 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2e8539-925f-4399-9379-be96886dedc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:47:44.001350 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2e8539-925f-4399-9379-be96886dedc4 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:47:44.012649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1462/2922] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:47:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:47:53.949765 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=15 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:47:54.007233 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=15 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:48:32.587989 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1056753-ae89-4bc4-a36c-18c53a5eb5a9 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:48:32.597652 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1056753-ae89-4bc4-a36c-18c53a5eb5a9 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:48:32.601389 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:48:32.601981 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:48:32.602284 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:48:32.602555 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:48:32.604307 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:48:32.613697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:48:32.615156 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:48:32.623345 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1461/2923] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:48:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:48:39.106413 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] REQUEST_METHOD: `POST` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:48:39.106900 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:48:39.107038 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:48:39.126860 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.auth.core [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] MFA Rules not processed for user `450dcd00d2404852b9635e72650d0c7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81399) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:48:39.128082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:48:39.182620 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1463/2924] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:48:39 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:48:42.619561 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:48:49.175207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=25 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:49:27.783801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:27.793102 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d5d656-4289-49a0-83b1-49d079512e0b service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:27.796639 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:27.797217 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:27.797488 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:27.797753 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:27.799469 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:27.858379 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:27.859897 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:27.868682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1462/2925] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:49:27 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:49:28.697603 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:28.699468 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:28.721394 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0d2e1a-0aaa-45db-88df-e69d694f228c None None] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:28.727451 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:28.728153 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:28.728493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:28.728830 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:28.730640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:28.743937 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:28.745502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:28.757505 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1464/2926] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:49:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:49:32.595418 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:32.604015 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c5f505-90c6-4ad6-8946-02b2274e1a06 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:32.607633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:32.608161 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:32.608435 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:32.608704 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:32.610410 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:32.672055 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:32.673520 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:32.683697 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1463/2927] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:49:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:49:36.014795 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:36.016363 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:36.093147 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee273ca3-05aa-42e4-8a7b-43906138ad12 service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:36.098624 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:36.099516 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:36.099832 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:36.100043 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:36.102048 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:36.114279 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:36.116397 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:36.127036 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1465/2928] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:49:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:49:39.151053 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:39.152614 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:39.167886 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1972b21e-c31a-4c9e-9062-175b2c8b30d4 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:39.172882 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:39.173527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:39.173762 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:39.174171 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:39.175891 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:39.189029 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:39.190608 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:39.202548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1464/2929] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:49:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:49:39.211520 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:39.213672 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:39.238684 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9075bea-c1da-4d17-a0ed-4968f8abc333 service placement] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:39.244793 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:39.245896 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:39.247437 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:39.247731 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] PATH_INFO: `/v3/users/040bc65ba7f94c14999a98266ff0ce7f` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:39.255789 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=040bc65ba7f94c14999a98266ff0ce7f)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:39.258328 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:39.308519 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '040bc65ba7f94c14999a98266ff0ce7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.309208 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '040bc65ba7f94c14999a98266ff0ce7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.316416 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '040bc65ba7f94c14999a98266ff0ce7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.329029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '040bc65ba7f94c14999a98266ff0ce7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.329695 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '040bc65ba7f94c14999a98266ff0ce7f'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.343353 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1466/2930] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:49:39 2022] DELETE /identity/v3/users/040bc65ba7f94c14999a98266ff0ce7f => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:49:39.510383 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:39.522739 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a984abf-e011-4adf-8e4e-97be98309f3f service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:39.580537 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:39.581164 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:39.581446 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:39.581721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] PATH_INFO: `/v3/projects/0a6698bc3af34041bd01a5d5abefb11c` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:39.582823 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a6698bc3af34041bd01a5d5abefb11c)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:39.585822 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:39.608906 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=100 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:49:39.631794 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a6698bc3af34041bd01a5d5abefb11c', 'request_id': 'req-ff54545c-77a6-4342-9507-4588e6832e7c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.637224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a6698bc3af34041bd01a5d5abefb11c', 'request_id': 'req-ff54545c-77a6-4342-9507-4588e6832e7c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.643322 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a6698bc3af34041bd01a5d5abefb11c', 'request_id': 'req-ff54545c-77a6-4342-9507-4588e6832e7c'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.655955 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] The token cache is being invalidate because project 0a6698bc3af34041bd01a5d5abefb11c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:49:39.656241 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.656853 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.658303 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1465/2931] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:49:39 2022] DELETE /identity/v3/projects/0a6698bc3af34041bd01a5d5abefb11c => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:49:39.663074 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:39.664458 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:39.678174 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eced2e08-d756-49e6-8e61-b6efd449c9f3 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:39.722527 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:49:39.736214 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:39.736835 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:39.737115 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:39.737385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] PATH_INFO: `/v3/users/b466a7db4d11416fbba073555a210d0d` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:39.738500 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b466a7db4d11416fbba073555a210d0d)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:39.741013 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:39.786109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b466a7db4d11416fbba073555a210d0d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.786671 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b466a7db4d11416fbba073555a210d0d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.794241 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b466a7db4d11416fbba073555a210d0d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.806700 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b466a7db4d11416fbba073555a210d0d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.807342 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b466a7db4d11416fbba073555a210d0d'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:39.820298 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1467/2932] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:49:39 2022] DELETE /identity/v3/users/b466a7db4d11416fbba073555a210d0d => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:49:39.999152 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:49:40.000643 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:49:40.015970 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff54545c-77a6-4342-9507-4588e6832e7c admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:49:40.075633 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:49:40.076252 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:49:40.076522 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:49:40.076787 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] PATH_INFO: `/v3/projects/794a333acb874a3784e504154d321157` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:49:40.077870 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=794a333acb874a3784e504154d321157)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:49:40.079790 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:49:40.125444 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '794a333acb874a3784e504154d321157', 'request_id': 'req-5633acdf-9499-42aa-a8be-f1a455be1b48'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:40.131541 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '794a333acb874a3784e504154d321157', 'request_id': 'req-5633acdf-9499-42aa-a8be-f1a455be1b48'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:40.137577 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '794a333acb874a3784e504154d321157', 'request_id': 'req-5633acdf-9499-42aa-a8be-f1a455be1b48'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:40.149770 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] The token cache is being invalidate because project 794a333acb874a3784e504154d321157 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81400) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:49:40.150049 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:40.150663 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-5633acdf-9499-42aa-a8be-f1a455be1b48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:49:40.152117 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1466/2933] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:49:39 2022] DELETE /identity/v3/projects/794a333acb874a3784e504154d321157 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 06 04:49:49.816351 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=31,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:49:50.146427 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:DELETE=14,keystone:SELECT=66,keystone:INSERT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:50:49.346873 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:50:49.348346 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:50:49.460091 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644b3988-3063-4dde-8eb5-2e8aa0dd7e05 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:50:49.509385 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '61d1b78e81db4dbe987d788fd398e89d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:50:49.509906 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:50:49.510183 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:50:49.510484 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:50:49.512307 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:50:49.581018 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:50:49.582421 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:50:49.591027 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1468/2934] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:50:49 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:50:59.577419 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] REQUEST_METHOD: `POST` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:50:59.577935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:50:59.577935 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:50:59.587021 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=96 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:50:59.597536 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.auth.core [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] MFA Rules not processed for user `450dcd00d2404852b9635e72650d0c7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81400) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 06 04:50:59.598721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:50:59.606220 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1467/2935] 10.222.0.33 () {58 vars in 958 bytes} [Thu Oct 6 04:50:59 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 06 04:50:59.924493 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:50:59.925862 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:00.001749 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a6b4d8-ad94-4c06-80b4-54101dc8548a service cinder] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:51:00.005274 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:51:00.005858 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:51:00.006130 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:51:00.006395 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:51:00.008082 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:00.017382 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:51:00.018771 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:51:00.026649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1469/2936] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:50:59 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:51:00.478139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:51:00.479767 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:00.579996 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02157178-0b28-4e64-bd60-61c0b431bde0 None None] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:51:00.585754 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:51:00.586386 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:51:00.586682 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:51:00.586988 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:51:00.588681 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:00.601139 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:51:00.602721 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:51:00.613057 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1468/2937] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:51:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:51:10.023109 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=35 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:51:10.608801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=47 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:51:12.868227 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:51:12.869655 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:12.944839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd3ce4-e579-4636-b486-3cc19a14a84b service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:51:12.950632 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c8d2db883de44964b6e15b85a97726b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'ResellerAdmin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:51:12.951262 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:51:12.951539 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:51:12.951809 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:51:12.953502 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:12.965814 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:51:12.967331 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:51:12.977649 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1470/2938] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:51:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 06 04:51:22.973204 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=34 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:51:41.889527 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:51:41.891010 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:41.903758 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a64544-b75b-4762-9c32-3476855c5fff service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:51:41.908677 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:51:41.909257 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:51:41.909529 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:51:41.909801 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:51:41.911476 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:51:41.923651 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:51:41.925224 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:51:41.935756 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1469/2939] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:51:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:51:51.930805 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:53:20.878705 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:53:20.890869 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9de2b56-909a-4f3b-90cb-fe350abde647 service nova] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:53:20.894995 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:53:20.895876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:53:20.895876 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:53:20.896123 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:53:20.897742 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:53:20.937457 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:53:20.938895 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:53:20.964392 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1471/2940] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:53:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:53:22.172862 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:53:22.174374 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:53:22.187346 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133577bd-bc72-4438-8dd6-034f8fc052ec service placement] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:53:22.190885 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:53:22.191540 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:53:22.191815 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:53:22.192088 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:53:22.193783 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:53:22.208458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:53:22.209201 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:53:22.222153 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1470/2941] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:53:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:53:30.961236 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=18 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:53:32.216204 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=6 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:53:33.053000 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:53:33.072929 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf97f62a-a329-46ec-8c6f-0c206aae3fdc service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:53:33.077287 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87829678318948f3b81bd107076bdfa7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:53:33.077898 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:53:33.078178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:53:33.078626 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:53:33.080414 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:53:33.175718 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:53:33.177156 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:53:33.197450 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1472/2942] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:53:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:53:39.312245 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:53:39.317225 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:53:39.344553 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef619bb6-ec22-4695-babe-e31a76eea596 service neutron] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:53:39.353310 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:53:39.354533 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:53:39.354807 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:53:39.355151 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] PATH_INFO: `/v3/users/450dcd00d2404852b9635e72650d0c7f` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:53:39.356302 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] RBAC: Authorizing `identity:delete_user(user_id=450dcd00d2404852b9635e72650d0c7f)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:53:39.358237 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:53:39.405144 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '450dcd00d2404852b9635e72650d0c7f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.405793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '450dcd00d2404852b9635e72650d0c7f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.412694 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '450dcd00d2404852b9635e72650d0c7f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.413306 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '450dcd00d2404852b9635e72650d0c7f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.425487 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '450dcd00d2404852b9635e72650d0c7f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.437122 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1471/2943] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:53:39 2022] DELETE /identity/v3/users/450dcd00d2404852b9635e72650d0c7f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:53:39.608854 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:53:39.623428 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b727a5-bce1-4cfb-86d4-b0a57f8105b2 service neutron] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:53:39.684012 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:53:39.684628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:53:39.684903 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:53:39.685172 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] PATH_INFO: `/v3/projects/a707ca1042dd4fc99ff1dbd033efe0e8` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:53:39.686257 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a707ca1042dd4fc99ff1dbd033efe0e8)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:53:39.688207 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:53:39.733640 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a707ca1042dd4fc99ff1dbd033efe0e8', 'request_id': 'req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.739610 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a707ca1042dd4fc99ff1dbd033efe0e8', 'request_id': 'req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.745931 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a707ca1042dd4fc99ff1dbd033efe0e8', 'request_id': 'req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.749944 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:53:39.759158 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] The token cache is being invalidate because project a707ca1042dd4fc99ff1dbd033efe0e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:53:39.759802 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.760780 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.762282 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1473/2944] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:53:39 2022] DELETE /identity/v3/projects/a707ca1042dd4fc99ff1dbd033efe0e8 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:53:39.766349 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:53:39.767776 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:53:39.780526 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca1a9fc-58de-4f04-94d7-d8c59f6f5702 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:53:39.837080 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:53:39.837701 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:53:39.838238 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:53:39.838458 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] PATH_INFO: `/v3/users/0fcbe8f22f6340f68db571f9ea06c21f` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:53:39.839589 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fcbe8f22f6340f68db571f9ea06c21f)` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:53:39.841494 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:53:39.887548 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fcbe8f22f6340f68db571f9ea06c21f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.888194 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fcbe8f22f6340f68db571f9ea06c21f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.895059 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fcbe8f22f6340f68db571f9ea06c21f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.895695 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fcbe8f22f6340f68db571f9ea06c21f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.908521 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.notifications [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fcbe8f22f6340f68db571f9ea06c21f'} {{(pid=81400) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:39.910184 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:53:39.922964 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1472/2945] 10.222.0.33 () {58 vars in 1224 bytes} [Thu Oct 6 04:53:39 2022] DELETE /identity/v3/users/0fcbe8f22f6340f68db571f9ea06c21f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:53:40.104670 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Authenticating user token {{(pid=81399) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:53:40.106090 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.fernet_utils [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81399) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:53:40.122178 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1e111d-bf93-4572-bbe0-9bcaadb755f3 admin admin] Validating token access rules against request {{(pid=81399) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:53:40.179839 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6cec6cd063d4a0684ad9f283f12b406', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7760d317949542fd9d86e529eca476cc', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81399) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:53:40.180489 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:53:40.180762 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:53:40.181029 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] PATH_INFO: `/v3/projects/31adbae4cbd54ad0992fdaeb77ea0e56` {{(pid=81399) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:53:40.182175 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=31adbae4cbd54ad0992fdaeb77ea0e56)` {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:53:40.184118 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] RBAC: Authorization granted {{(pid=81399) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:53:40.230197 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31adbae4cbd54ad0992fdaeb77ea0e56', 'request_id': 'req-65404d02-9de8-4312-8274-1cb3aff3c6a8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:40.236699 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31adbae4cbd54ad0992fdaeb77ea0e56', 'request_id': 'req-65404d02-9de8-4312-8274-1cb3aff3c6a8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:40.242343 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31adbae4cbd54ad0992fdaeb77ea0e56', 'request_id': 'req-65404d02-9de8-4312-8274-1cb3aff3c6a8'} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:40.255767 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] The token cache is being invalidate because project 31adbae4cbd54ad0992fdaeb77ea0e56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81399) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 06 04:53:40.255989 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:40.256628 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG keystone.notifications [None req-65404d02-9de8-4312-8274-1cb3aff3c6a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81399) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 06 04:53:40.258104 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: [pid: 81399|app: 0|req: 1474/2946] 10.222.0.33 () {58 vars in 1233 bytes} [Thu Oct 6 04:53:40 2022] DELETE /identity/v3/projects/31adbae4cbd54ad0992fdaeb77ea0e56 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 06 04:53:49.920231 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=2 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:53:50.252543 dv-cinder-860227-2-iscsi devstack@keystone.service[81399]: DEBUG dbcounter [-] [81399] Writing DB stats keystone:SELECT=53,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81399) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 06 04:56:50.818202 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Authenticating user token {{(pid=81400) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 06 04:56:50.819637 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:56:50.890778 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c0fa78-d06f-4583-ab22-36485ded06a3 admin admin] Validating token access rules against request {{(pid=81400) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 06 04:56:50.951211 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6390b02-fbb3-4581-b241-a8a30098c535 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7e2fe69b3b3b4ba5b55a4698b3ed70b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dd80852f67f74db3be0a04ef0a28ec38', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81400) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 06 04:56:50.951820 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6390b02-fbb3-4581-b241-a8a30098c535 service placement] REQUEST_METHOD: `GET` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 06 04:56:50.952101 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6390b02-fbb3-4581-b241-a8a30098c535 service placement] SCRIPT_NAME: `/identity` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 06 04:56:50.952377 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6390b02-fbb3-4581-b241-a8a30098c535 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81400) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 06 04:56:50.954247 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.fernet_utils [None req-d6390b02-fbb3-4581-b241-a8a30098c535 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81400) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 06 04:56:50.965530 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6390b02-fbb3-4581-b241-a8a30098c535 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 06 04:56:50.967793 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6390b02-fbb3-4581-b241-a8a30098c535 service placement] RBAC: Authorization granted {{(pid=81400) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 06 04:56:50.980216 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: [pid: 81400|app: 0|req: 1473/2947] 10.222.0.33 () {60 vars in 1365 bytes} [Thu Oct 6 04:56:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 06 04:57:00.975550 dv-cinder-860227-2-iscsi devstack@keystone.service[81400]: DEBUG dbcounter [-] [81400] Writing DB stats keystone:SELECT=58 {{(pid=81400) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}